Individual in multimodal integration patterns: what are they and why do they exist?

Sharon Oviatt, Rebecca Lunsford, Rachel Coulston

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

40 Citations (Scopus)


Techniques for information fusion are at the heart of multimodal system design. To develop new user-adaptive approaches for multimodal fusion, the present research investigated the stability and underlying cause of major individual differences that have been documented between users in their multimodal integration pattern. Longitudinal data were collected from 25 adults as they interacted with a map system over six weeks. Analyses of 1,100 multimodal constructions revealed that everyone had a dominant integration pattern, either simultaneous or sequential, which was 95-96% consistent and remained stable over time. In addition, coherent behavioral and linguistic differences were identified between these two groups. Whereas performance speed was comparable, sequential integrators made only half as many errors and excelled during new or complex tasks. Sequential integrators also had more precise articulation (e.g., fewer disfluencies), although their speech rate was no slower. Finally, sequential integrators more often adopted terse and direct command-style language, with a smaller and less varied vocabulary, which appeared focused on achieving error-free communication. These distinct interaction patterns are interpreted as deriving from fundamental differences in reflective-impulsive cognitive style. Implications of these findings are discussed for the design of adaptive multimodal systems with substantially improved performance characteristics.

Original languageEnglish
Title of host publicationCHI '05 - Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
EditorsW. Kellogg, S. Zhai, C. Gale, G. Veer
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
ISBN (Electronic)1581139985
ISBN (Print)9781581139983
Publication statusPublished - 2005
Externally publishedYes
EventInternational Conference on Human Factors in Computing Systems 2005 - Portland, United States of America
Duration: 2 Apr 20057 Apr 2005
Conference number: 23rd


ConferenceInternational Conference on Human Factors in Computing Systems 2005
Abbreviated titleCHI 2005
CountryUnited States of America


  • Commands
  • Conversations
  • Disfluencies
  • Errors
  • Impulsive-reflective cognitive style
  • Individual differences
  • Multimodal integration patterns
  • Simultaneous or sequential input

Cite this