When do we interact multimodally? cognitive load and multimodal communication patterns

Sharon Oviatt, Rachel Coulston, Rebecca Lunsford

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

205 Citations (Scopus)


Mobile usage patterns often entail high and fluctuating levels of difficulty as well as dual tasking. One major theme explored in this research is whether a flexible multimodal interface supports users in managing cognitive load. Findings from this study reveal that multimodal interface users spontaneously respond to dynamic changes in their own cognitive load by shifting to multimodal communication as load increases with task difficulty and communicative complexity. Given a flexible multimodal interface, users' ratio of multimodal (versus unimodal) interaction increased substantially from 18.6% when referring to established dialogue context to 77.1% when required to establish a new context, a +315% relative increase. Likewise, the ratio of users' multimodal interaction increased significantly as the tasks became more difficult, from 59.2% during low difficulty tasks, to 65.5% at moderate difficulty, 68.2% at high and 75.0% at very high difficulty, an overall relative increase of +27%. Analysis of users' task-critical errors and response latencies across task difficulty levels increased systematically and significantly as well, corroborating the manipulation of cognitive processing load. The adaptations seen in this study reflect users' efforts to self-manage limitations on working memory when task complexity increases. This is accomplished by distributing communicative information across multiple modalities, which is compatible with a cognitive load theory of multimodal interaction. The long-term goal of this research is the development of an empirical foundation for proactively guiding flexible and adaptive multimodal system design.

Original languageEnglish
Title of host publicationICMI'04 - Sixth International Conference on Multimodal Interfaces
Subtitle of host publication14 October 2004 through 15 October 2004
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages8
ISBN (Print)1581139543
Publication statusPublished - 2004
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2004 - State College, United States of America
Duration: 13 Oct 200415 Oct 2004
Conference number: 6th
https://dl.acm.org/doi/proceedings/10.1145/1027933 (Proceedings)


ConferenceInternational Conference on Multimodal Interfaces 2004
Abbreviated titleICMI 2004
Country/TerritoryUnited States of America
CityState College
Internet address


  • Cognitive load
  • Dialogue context
  • Human performance
  • Individual differences
  • Multimodal integration
  • Multimodal interaction
  • Speech and pen input
  • System adaptation
  • Task difficulty
  • Unimodal interaction

Cite this