Multimodal interaction under exerted conditions in a natural field setting

Sanjeev Kumar, Philip R. Cohen, Rachel Coulston

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

8 Citations (Scopus)


This paper evaluates the performance of a multimodal interface under exerted conditions in a natural field setting. The subjects in the present study engaged in a strenuous activity while multimodally performing map-based tasks using handheld computing devices. This activity made the users breathe heavily and become fatigued during the course of the study. We found that the performance of both speech and gesture recognizers degraded as a function of exertion, while the overall multimodal success rate was stable. This stabilization is accounted for by the mutual disambiguation of modalities, which increases significantly with exertion. The system performed better for subjects with a greater level of physical fitness, as measured by their running speed, with more stable multimodal performance and a later degradation of speech and gesture recognition as compared with subjects who were less fit. The findings presented in this paper have a significant impact on design decisions for multimodal interfaces targeted towards highly mobile and exerted users in field environments.

Original languageEnglish
Title of host publicationICMI'04 - Sixth International Conference on Multimodal Interfaces
Number of pages8
Publication statusPublished - 1 Dec 2004
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2004 - State College, United States of America
Duration: 13 Oct 200415 Oct 2004
Conference number: 6th (Proceedings)

Publication series

NameICMI'04 - Sixth International Conference on Multimodal Interfaces


ConferenceInternational Conference on Multimodal Interfaces 2004
Abbreviated titleICMI 2004
Country/TerritoryUnited States of America
CityState College
Internet address


  • Evaluation
  • Exertion
  • Field
  • Mobile
  • Multimodal Interaction

Cite this