A multimodal system to characterise melancholia: cascaded bag of words approach

Shalini Bhatia, Munawar Hayat, Roland Goecke

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

3 Citations (Scopus)


Recent years have seen a lot of activity in affective computing for automated analysis of depression. However, no research has so far proposed a multimodal system for classifying different subtypes of depression such as melancholia. The mental state assessment of a mood disorder depends primarily on appearance, behaviour, speech, thought, perception, mood and facial affect. Mood and facial affect mainly contribute to distinguishing melancholia from nonmelancholia. These are assessed by clinicians, and hence vulnerable to subjective judgement. As a result, clinical assessment alone may not accurately capture the presence or absence of specific disorders such as melancholia, a distressing condition whose presence has important treatment implications. Melancholia is characterised by severe anhedonia and psychomotor disturbance, which can be a combination of motor retardation with periods of superimposed agitation. Psychomotor disturbance can be sensed in both face and voice. To the best of our knowledge, this study is the first attempt to propose a multimodal system to differentiate melancholia from non-melancholia and healthy controls. We report the sensitivity and specificity of classification in depressive subtypes.

Original languageEnglish
Title of host publicationICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction
EditorsEdward Lank, Alessandro Vinciarelli, Eve Hoggan, Sriram Subramanian, Stephen A. Brewster
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages7
ISBN (Electronic)9781450355438
Publication statusPublished - Nov 2017
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2017 - Glasgow, United Kingdom
Duration: 13 Nov 201717 Nov 2017
Conference number: 19th
https://icmi.acm.org/2017/ (conference website)
https://dl.acm.org/doi/proceedings/10.1145/3136755 (Proceedings)


ConferenceInternational Conference on Multimodal Interfaces 2017
Abbreviated titleICMI 2017
Country/TerritoryUnited Kingdom
Internet address


  • Audio
  • Bag of words
  • Fusion
  • Multimodal
  • Video

Cite this