Prediction of heart rate changes from speech features during interaction with a misbehaving dialog system

Andreas Tsiartas, Andreas Kathol, Elizabeth Shriberg, Massimiliano De Zambotti, Adrian Willoughby

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

4 Citations (Scopus)

Abstract

Most research on detecting a speaker's cognitive state when interacting with a dialog system has been based on selfreports, or on hand-coded subjective judgments based on audio or audio-visual observations. This study examines two questions: (1) how do undesirable system responses affect people physiologically, and (2) to what extent can we predict physiological changes from the speech signal alone? To address these questions, we use a new corpus of simultaneous speech and high-quality physiological recordings in the product returns domain (the SRI BioFrustration Corpus). "Triggers" were used to frustrate users at specific times during the interaction to produce emotional responses at similar times during the experiment across participants. For each of eight return tasks per participant, we compared speaker-normalized pre-trigger (cooperative system behavior) regions to posttrigger (uncooperative system behavior) regions. Results using random forest classifiers show that changes in spectral and temporal features of speech can predict heart rate changes with an accuracy of ~70%. Implications for future research and applications are discussed.

Original languageEnglish
Title of host publicationINTERSPEECH-2015
Subtitle of host publication16th Annual Conference of the International Speech Communication Association
Pages3715-3719
Number of pages5
Volume2015-January
Publication statusPublished - 2015
Externally publishedYes
EventAnnual Conference of the International Speech Communication Association INTERSPEECH 2015 - Dresden, Germany
Duration: 6 Sep 201510 Sep 2015
Conference number: 16th
http://interspeech2015.org/
https://dblp.org/db/conf/interspeech/interspeech2015.html (Proceedings)

Conference

ConferenceAnnual Conference of the International Speech Communication Association INTERSPEECH 2015
Abbreviated titleInterspeech 2015
Country/TerritoryGermany
CityDresden
Period6/09/1510/09/15
Internet address

Keywords

  • Autonomic nervous system
  • Dialog systems
  • Emotion
  • Frustration
  • Heart rate
  • Speech corpora
  • Speech features

Cite this