Multimodal analysis and estimation of intimate self-disclosure

Mohammad Soleymani, Kalin Stefanov, Sin Hwa Kang, Jan Ondras, Jonathan Gratch

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)

Abstract

Self-disclosure to others has a proven benefit for one's mental health. It is shown that disclosure to computers can be similarly beneficial for emotional and psychological well-being. In this paper, we analyzed verbal and nonverbal behavior associated with self-disclosure in two datasets containing structured human-human and human-agent interviews from more than 200 participants. Correlation analysis of verbal and nonverbal behavior revealed that linguistic features such as affective and cognitive content in verbal behavior, and nonverbal behavior such as head gestures are associated with intimate self-disclosure. A multimodal deep neural network was developed to automatically estimate the level of intimate self-disclosure from verbal and nonverbal behavior. Between modalities, verbal behavior was the best modality for estimating self-disclosure within-corpora achieving r = 0.66. However, the cross-corpus evaluation demonstrated that nonverbal behavior can outperform language modality in cross-corpus evaluation. Such automatic models can be deployed in interactive virtual agents or social robots to evaluate rapport and guide their conversational strategy.

Original languageEnglish
Title of host publicationProceedings of the 2019 International Conference onMultimodal Interaction
EditorsSusan R. Fussell, Bjorn Schuller, Yale Song, Kai Yu
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages59-68
Number of pages10
ISBN (Electronic)9781450368605
DOIs
Publication statusPublished - 2019
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2019 - Suzhou, China
Duration: 14 Oct 201918 Oct 2019
Conference number: 21st
http://icmi.acm.org/2019/
https://dl.acm.org/doi/proceedings/10.1145/3340555 (Proceedings)

Conference

ConferenceInternational Conference on Multimodal Interfaces 2019
Abbreviated titleICMI 2019
CountryChina
CitySuzhou
Period14/10/1918/10/19
Internet address

Keywords

  • Natural language understanding
  • Neural networks
  • Nonverbal behavior
  • Self-disclosure

Cite this