Multimodal learning for identifying opportunities for empathetic responses

Leili Tavabi, Kalin Stefanov, Setareh Nasihati Gilani, David Traum, Mohammad Soleymani

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

13 Citations (Scopus)


Embodied interactive agents possessing emotional intelligence and empathy can create natural and engaging social interactions. Providing appropriate responses by interactive virtual agents requires the ability to perceive users' emotional states. In this paper, we study and analyze behavioral cues that indicate an opportunity to provide an empathetic response. Emotional tone in language in addition to facial expressions are strong indicators of dramatic sentiment in conversation that warrant an empathetic response. To automatically recognize such instances, we develop a multimodal deep neural network for identifying opportunities when the agent should express positive or negative empathetic responses. We train and evaluate our model using audio, video and language from human-agent interactions in a wizard-of-Oz setting, using the wizard's empathetic responses and annotations collected on Amazon Mechanical Turk as ground-truth labels. Our model outperforms a textbased baseline achieving F1-score of 0.71 on a three-class classification.We further investigate the results and evaluate the capability of such a model to be deployed for real-world human-agent interactions.

Original languageEnglish
Title of host publicationProceedings of the 2019 International Conference onMultimodal Interaction
EditorsSusan R. Fussell, Bjorn Schuller, Yale Song, Kai Yu
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages10
ISBN (Electronic)9781450368605
Publication statusPublished - 2019
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2019 - Suzhou, China
Duration: 14 Oct 201918 Oct 2019
Conference number: 21st (Proceedings)


ConferenceInternational Conference on Multimodal Interfaces 2019
Abbreviated titleICMI 2019
Internet address


  • Empathy
  • Human behavior
  • Machine learning
  • Multimodal sentiment
  • Virtual human

Cite this