Emotion or expressivity? An automated analysis of nonverbal perception in a social dilemma

Su Lei, Kalin Stefanov, Jonathan Gratch

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


An extensive body of research has examined how specific emotional expressions shape social perceptions and social decisions, yet recent scholarship in emotion research has raised questions about the validity of emotion as a construct. In this article, we contrast the value of measuring emotional expressions with the more general construct of expressivity (in the sense of conveying a thought or emotion through any nonverbal behavior) and develop models that can automatically extract perceived expressivity from videos. Although less extensive, a solid body of research has shown expressivity to be an important element when studying interpersonal perception, particularly in psychiatric contexts. Here we examine the role expressivity plays in predicting social perceptions and decisions in the context of a social dilemma. We show that perceivers use more than facial expressions when making judgments of expressivity and see these expressions as conveying thoughts as well as emotions (although facial expressions and emotional attributions explain most of the variance in these judgments). We next show that expressivity can be predicted with high accuracy using Lasso and random forests. Our analysis shows that features related to motion dynamics are particularly important for modeling these judgments. We also show that learned models of expressivity have value in recognizing important aspects of a social situation. First, we revisit a previously published finding which showed that smile intensity was associated with the unexpectedness of outcomes in social dilemmas; instead, we show that expressivity is a better predictor (and explanation) of this finding. Second, we provide preliminary evidence that expressivity is useful for identifying “moments of interest” in a video sequence.
Original languageEnglish
Title of host publicationProceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition
EditorsVitomir Štruc, Francisco Gómez-Fernández
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages8
ISBN (Electronic)9781728130798
ISBN (Print)9781728130804
Publication statusPublished - 2020
Externally publishedYes
EventIEEE International Conference on Automatic Face and Gesture Recognition 2020 - Buenos Aires, Argentina
Duration: 16 Nov 202020 Nov 2020
Conference number: 15th
https://ieeexplore-ieee-org.ezproxy.lib.monash.edu.au/xpl/conhome/9320148/proceeding (Proceedings)
https://fg2020.org (Website)


ConferenceIEEE International Conference on Automatic Face and Gesture Recognition 2020
Abbreviated titleFG 2020
CityBuenos Aires
Internet address

Cite this