Integration and synchronization of input modes during multimodal human-computer interaction

Sharon Oviatt, Antonella DeAngeli, Karen Kuhn

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

202 Citations (Scopus)

Abstract

Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system. Task analysis revealed that multimodal interaction occurred most frequently during spatial location commands, and with intermediate frequency during selection commands. In addition, microanalysis of input signals identified sequential, simultaneous, point-and-speak, and compound integration patterns, as well as data on the temporal precedence of modes and on inter-modal lags. In synchronizing input streams, the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence. Linguistic analysis also revealed that the spoken and written modes consistently supplied complementary semantic information, rather than redundant. One long-term goal of this research is the development of predictive models of natural modality integration to guide the design of emerging multimodal architectures.

Original languageEnglish
Title of host publicationCHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Subtitle of host publicationAtlanta, Georgia, USA — March 22 - 27, 1997
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages415-422
Number of pages8
ISBN (Electronic)0897918029
Publication statusPublished - 1997
Externally publishedYes
EventInternational Conference on Human Factors in Computing Systems 1997 - Atlanta, United States of America
Duration: 22 Mar 199727 Mar 1997
Conference number: 15th

Conference

ConferenceInternational Conference on Human Factors in Computing Systems 1997
Abbreviated titleCHI 1997
CountryUnited States of America
CityAtlanta
Period22/03/9727/03/97

Cite this

Oviatt, S., DeAngeli, A., & Kuhn, K. (1997). Integration and synchronization of input modes during multimodal human-computer interaction. In CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems: Atlanta, Georgia, USA — March 22 - 27, 1997 (pp. 415-422). New York NY USA: Association for Computing Machinery (ACM).
Oviatt, Sharon ; DeAngeli, Antonella ; Kuhn, Karen. / Integration and synchronization of input modes during multimodal human-computer interaction. CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems: Atlanta, Georgia, USA — March 22 - 27, 1997 . New York NY USA : Association for Computing Machinery (ACM), 1997. pp. 415-422
@inproceedings{074589cd9b7f4836a88825f532833c30,
title = "Integration and synchronization of input modes during multimodal human-computer interaction",
abstract = "Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system. Task analysis revealed that multimodal interaction occurred most frequently during spatial location commands, and with intermediate frequency during selection commands. In addition, microanalysis of input signals identified sequential, simultaneous, point-and-speak, and compound integration patterns, as well as data on the temporal precedence of modes and on inter-modal lags. In synchronizing input streams, the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence. Linguistic analysis also revealed that the spoken and written modes consistently supplied complementary semantic information, rather than redundant. One long-term goal of this research is the development of predictive models of natural modality integration to guide the design of emerging multimodal architectures.",
author = "Sharon Oviatt and Antonella DeAngeli and Karen Kuhn",
year = "1997",
language = "English",
pages = "415--422",
booktitle = "CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems",
publisher = "Association for Computing Machinery (ACM)",
address = "United States of America",

}

Oviatt, S, DeAngeli, A & Kuhn, K 1997, Integration and synchronization of input modes during multimodal human-computer interaction. in CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems: Atlanta, Georgia, USA — March 22 - 27, 1997 . Association for Computing Machinery (ACM), New York NY USA, pp. 415-422, International Conference on Human Factors in Computing Systems 1997, Atlanta, United States of America, 22/03/97.

Integration and synchronization of input modes during multimodal human-computer interaction. / Oviatt, Sharon; DeAngeli, Antonella; Kuhn, Karen.

CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems: Atlanta, Georgia, USA — March 22 - 27, 1997 . New York NY USA : Association for Computing Machinery (ACM), 1997. p. 415-422.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Integration and synchronization of input modes during multimodal human-computer interaction

AU - Oviatt, Sharon

AU - DeAngeli, Antonella

AU - Kuhn, Karen

PY - 1997

Y1 - 1997

N2 - Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system. Task analysis revealed that multimodal interaction occurred most frequently during spatial location commands, and with intermediate frequency during selection commands. In addition, microanalysis of input signals identified sequential, simultaneous, point-and-speak, and compound integration patterns, as well as data on the temporal precedence of modes and on inter-modal lags. In synchronizing input streams, the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence. Linguistic analysis also revealed that the spoken and written modes consistently supplied complementary semantic information, rather than redundant. One long-term goal of this research is the development of predictive models of natural modality integration to guide the design of emerging multimodal architectures.

AB - Our ability to develop robust multimodal systems will depend on knowledge of the natural integration patterns that typify people's combined use of different input modes. To provide a foundation for theory and design, the present research analyzed multimodal interaction while people spoke and wrote to a simulated dynamic map system. Task analysis revealed that multimodal interaction occurred most frequently during spatial location commands, and with intermediate frequency during selection commands. In addition, microanalysis of input signals identified sequential, simultaneous, point-and-speak, and compound integration patterns, as well as data on the temporal precedence of modes and on inter-modal lags. In synchronizing input streams, the temporal precedence of writing over speech was a major theme, with pen input conveying location information first in a sentence. Linguistic analysis also revealed that the spoken and written modes consistently supplied complementary semantic information, rather than redundant. One long-term goal of this research is the development of predictive models of natural modality integration to guide the design of emerging multimodal architectures.

UR - http://www.scopus.com/inward/record.url?scp=0030646107&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:0030646107

SP - 415

EP - 422

BT - CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

ER -

Oviatt S, DeAngeli A, Kuhn K. Integration and synchronization of input modes during multimodal human-computer interaction. In CHI '97 Proceedings of the ACM SIGCHI Conference on Human factors in computing systems: Atlanta, Georgia, USA — March 22 - 27, 1997 . New York NY USA: Association for Computing Machinery (ACM). 1997. p. 415-422