Integration and synchronization of input modes during multimodal human-computer interaction

Sharon Oviatt, Antonella DeAngeli, Karen Kuhn

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

168 Citations (Scopus)


Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system. Task analysis revealed that multimodal interaction occurred most frequently during spatial location commands, and with intermediate frequency during selection commands. In addition, microanalysis of input signals identified sequential, simultaneous, point-and-speak, and compound integration patterns, as well as data on the temporal precedence of modes and on inter-modal lags. In synchronizing input streams, the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence. Linguistic analysis also revealed that the spoken and written modes consistently supplied complementary semantic information, rather than redundant. One long-term goal of this research is the development of predictive models of natural modality integration to guide the design of emerging multimodal architectures.

Original languageEnglish
Title of host publicationCHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Subtitle of host publicationAtlanta, Georgia, USA — March 22 - 27, 1997
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages8
ISBN (Electronic)0897918029
Publication statusPublished - 1997
Externally publishedYes
EventInternational Conference on Human Factors in Computing Systems 1997 - Atlanta, United States of America
Duration: 22 Mar 199727 Mar 1997
Conference number: 15th


ConferenceInternational Conference on Human Factors in Computing Systems 1997
Abbreviated titleCHI 1997
Country/TerritoryUnited States of America

Cite this