Investigating word affect features and fusion of probabilistic predictions incorporating uncertainty in AVEC 2017

Ting Dang, Mia Atcheson, Brian Stasak, Munawar Hayat, Roland Goecke, Zhaocheng Huang, Phu Le, Julien Epps, Sadari Jayawardena, Vidhyasaharan Sethu

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch

15 Citations (Scopus)


Predicting emotion intensity and severity of depression are both challenging and important problems within the broader field of affective computing. As part of the AVEC 2017, we developed a number of systems to accomplish these tasks. In particular, word affect features, which derive human affect ratings (e.g. arousal and valence) from transcripts, were investigated for predicting depression severity and liking, showing great promise. A simple system based on the word affect features achieved an RMSE of 6.02 on the test set, yielding a relative improvement of 13.6% over the baseline. For the emotion prediction sub-challenge, we investigated multimodal fusion, which incorporated a measure of uncertainty associated with each prediction within an Output-Associative fusion framework for arousal and valence prediction, whilst liking prediction systems mainly focused on text-based features. Our best emotion prediction systems provided significant relative improvements over the baseline on the test set of 39.5%, 17.6%, and 29.3% for arousal, valence, and liking. Of particular note is that consistent improvements were observed when incorporating prediction uncertainty across various system configurations for predicting arousal and valence, suggesting the importance of taking into consideration prediction uncertainty for fusion and more broadly the advantages of probabilistic predictions.

Original languageEnglish
Title of host publicationProceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge
EditorsFabien Ringeval, Björn Schuller, Michel Valstar, Jonathan Gratch, Roddy Cowie, Maja Pantic
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
ISBN (Electronic)9781450355025
Publication statusPublished - 2017
Externally publishedYes
EventAnnual Workshop on Audio/Visual Emotion Challenge 2017 - Mountain View, United States of America
Duration: 23 Oct 201723 Oct 2017
Conference number: 7th


ConferenceAnnual Workshop on Audio/Visual Emotion Challenge 2017
Abbreviated titleAVEC 2017
CountryUnited States of America
CityMountain View
Internet address


  • Depression prediction
  • Dimensional emotion prediction
  • Outputassociative fusion
  • Sentiment analysis
  • Uncertainty prediction

Cite this