Multimodal Input

Natalie Ruiz, Fang Chen, Sharon Oviatt

Research output: Chapter in Book/Report/Conference proceedingChapter (Book)Researchpeer-review

13 Citations (Scopus)


This chapter reviews the advantages of multimodal interfaces and presents some examples of state-of-the-art multimodal systems. The focus is on the links between multimodality and cognition, namely the application of human cognitive processing models to improve understanding of multimodal behavior in different contexts, particularly in situations of high mental demand. The growing interest in multimodal interface design is inspired largely by the goal of supporting more flexible, powerfully expressive, and low cognitive load means of human-computer interaction (HCI). They have the potential to expand computing to more mobile and complex real-world applications, to be used by a broader spectrum of everyday people and to accommodate more adverse usage conditions than in the past. The state-of-the-art multimodal input systems are currently only able to process two to three modal input channels, are application-specific, and have limited interaction vocabularies and grammars. The two most mature types of multimodal systems, in which the keyboard and mouse have been completely replaced, are those that combine either speech and pen input or speech and lip movements.

Original languageEnglish
Title of host publicationMultimodal Signal Processing
PublisherAcademic Press
Number of pages25
ISBN (Print)9780123748256
Publication statusPublished - 2010
Externally publishedYes

Cite this