Multimodal interfaces: a survey of principles, models and frameworks

Bruno Dumas, Denis Lalanne, Sharon Oviatt

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

196 Citations (Scopus)


The grand challenge of multimodal interface creation is to build reliable processing systems able to analyze and understand multiple communication means in real-time. This opens a number of associated issues covered by this chapter, such as heterogeneous data types fusion, architectures for real-time processing, dialog management, machine learning for multimodal interaction, modeling languages, frameworks, etc. This chapter does not intend to cover exhaustively all the issues related to multimodal interfaces creation and some hot topics, such as error handling, have been left aside. The chapter starts with the features and advantages associated with multimodal interaction, with a focus on particular findings and guidelines, as well as cognitive foundations underlying multimodal interaction. The chapter then focuses on the driving theoretical principles, time-sensitive software architectures and multimodal fusion and fission issues. Modeling of multimodal interaction as well as tools allowing rapid creation of multimodal interfaces are then presented. The article concludes with an outline of the current state of multimodal interaction research in Switzerland, and also summarizes the major future challenges in the field.

Original languageEnglish
Title of host publicationHuman Machine Interaction
Subtitle of host publicationResearch Results of the MMI Program
EditorsDenis Lalanne, Jurg Kohlas
Place of PublicationBerlin Germany
Number of pages24
ISBN (Electronic)9783642004377
ISBN (Print)3642004369, 9783642004360
Publication statusPublished - 2009
Externally publishedYes

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Cite this