Multimodal automatic coding of client behavior in Motivational Interviewing

Leili Tavabi, Kalin Stefanov, Larry Zhang, Brian Borsari, Joshua D. Woolley, Stefan Scherer, Mohammad Soleymani

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

13 Citations (Scopus)


Motivational Interviewing (MI) is defined as a collaborative conversation style that evokes the client's own intrinsic reasons for behavioral change. In MI research, the clients' attitude (willingness or resistance) toward change as expressed through language, has been identified as an important indicator of their subsequent behavior change. Automated coding of these indicators provides systematic and efficient means for the analysis and assessment of MI therapy sessions. In this paper, we study and analyze behavioral cues in client language and speech that bear indications of the client's behavior toward change during a therapy session, using a database of dyadic motivational interviews between therapists and clients with alcohol-related problems. Deep language and voice encoders, \ie BERT and VGGish, trained on large amounts of data are used to extract features from each utterance. We develop a neural network to automatically detect the MI codes using both the clients' and therapists' language and clients' voice, and demonstrate the importance of semantic context in such detection. Additionally, we develop machine learning models for predicting alcohol-use behavioral outcomes of clients through language and voice analysis. Our analysis demonstrates that we are able to estimate MI codes using clients' textual utterances along with preceding textual context from both the therapist and client, reaching an F1-score of 0.72 for a speaker-independent three-class classification. We also report initial results for using the clients' data for predicting behavioral outcomes, which outlines the direction for future work.

Original languageEnglish
Title of host publicationProceedings of the 2020 International Conference on Multimodal Interaction
EditorsNadia Berthouze, Mohamed Chetouani, Mikio Nakano
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages8
ISBN (Electronic)9781450375818
Publication statusPublished - 2020
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2020 - Virtual , Netherlands
Duration: 25 Oct 202029 Oct 2020
Conference number: 22nd (Website) (Proceedings)


ConferenceInternational Conference on Multimodal Interfaces 2020
Abbreviated titleICMI 2020
Internet address


  • human behavior
  • machine learning
  • mental health
  • motivational interviewing

Cite this