Analysis and prediction of heart rate using speech features from natural speech

Jennifer Smith, Andreas Tsiartas, Elizabeth Shriberg, Andreas Kathol, Adrian Willoughby, Massimiliano De Zambotti

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

9 Citations (Scopus)


Interactive voice technologies can leverage biosignals, such as heart rate (HR), to infer the psychophysiological state of the user. Voice-based detection of HR is attractive because it does not require additional sensors. We predict HR from speech using the SRI BioFrustration Corpus. In contrast to previous studies we use continuous spontaneous speech as input. Results using random forests show modest but significant effects on HR prediction. We further explore the effects on HR of speaking itself, and contrast the effects when interactions induce neutral versus frustrated responses from users. Results reveal that regardless of the user's emotional state, HR tends to increase while the user is engaged in speaking to a dialog system relative to a silent region right before speech, and that this effect is greater when the subject is expressing frustration. We also find that the user's HR does not recover to pre-speaking levels as quickly after frustrated speech as it does after neutral speech. Implications and future directions are discussed.

Original languageEnglish
Title of host publication2017 IEEE International Conference on Acoustics, Speech, and Signal Processing
Subtitle of host publicationProceedings
Place of PublicationUnited States
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages5
ISBN (Electronic)9781509041176
Publication statusPublished - 16 Jun 2017
Externally publishedYes
EventIEEE International Conference on Acoustics, Speech and Signal Processing 2017 - New Orleans, United States of America
Duration: 5 Mar 20179 Mar 2017 (Proceedings)

Publication series

NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
ISSN (Print)1520-6149


ConferenceIEEE International Conference on Acoustics, Speech and Signal Processing 2017
Abbreviated titleICASSP 2017
Country/TerritoryUnited States of America
CityNew Orleans
Internet address


  • autonomic nervous system
  • dialog system
  • frustration
  • heart rate
  • speech features

Cite this