Toward adaptive information fusion in multimodal systems

Xiao Huang, Sharon Oviatt

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

6 Citations (Scopus)


In recent years, a new generation of multimodal systems has emerged as a major direction within the HCI community. Multimodal interfaces and architectures are time-critical and data-intensive to develop, which poses new research challenges. The goal of the present work is to model and adapt to users' multimodal integration patterns, so that faster and more robust systems can be developed with on-line adaptation to individual's multimodal temporal thresholds. In this paper, we summarize past user-modeling results on speech and pen multimodal integration patterns, which indicate that there are two dominant types of multimodal integration pattern among users that can be detected very early and remain highly consistent. The empirical results also indicate that, when interacting with a multimodal system, users intermix unimodal with multimodal commands. Based on these results, we present new machine-learning results comparing three models of on-line system adaptation to users' integration patterns, which were based on Bayesian Belief Networks. This work utilized data from ten adults who provided approximately 1,000 commands while interacting with a map-based multimodal system. Initial experimental results with our learning models indicated that 85% of users' natural mixed input could be correctly classified as either unimodal or multimodal, and 82% of users' mulitmodal input could be correctly classified as either sequentially or simultaneously integrated. The long-term goal of this research is to develop new strategies for combining empirical user modeling with machine learning techniques to bootstrap accelerated, generalized, and improved reliability of information fusion in new types of multimodal system.

Original languageEnglish
Title of host publicationMachine Learning for Multimodal Interaction - Second International Workshop, MLMI 2005, Revised Selected Papers
Place of PublicationCham Switzerland
Number of pages13
ISBN (Print)3540325492, 9783540325499
Publication statusPublished - 2006
Externally publishedYes
Event2nd International Workshop on Machine Learning for Multimodal Interaction, MLMI 2005 - Edinburgh, United Kingdom
Duration: 11 Jul 200513 Jul 2005

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference2nd International Workshop on Machine Learning for Multimodal Interaction, MLMI 2005
Country/TerritoryUnited Kingdom

Cite this