Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Ed Kaiser, Alex Olwal, David McGee, Hrvoje Benko, Andrea Corradini, Xiaoguang Li, Phil Cohen, Steven Feiner

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

123 Citations (Scopus)


We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

Original languageEnglish
Title of host publicationICMI'03
Subtitle of host publicationFifth International Conference on Multimodal Interfaces
Number of pages8
Publication statusPublished - 1 Dec 2003
Externally publishedYes
EventInternational Conference on Multimodal Interfaces 2003 - Vancouver, Canada
Duration: 5 Nov 20037 Nov 2003
Conference number: 5th (Proceedings)

Publication series

NameICMI'03: Fifth International Conference on Multimodal Interfaces


ConferenceInternational Conference on Multimodal Interfaces 2003
Abbreviated titleICMI 2003
Internet address


  • Augmented/virtual reality
  • Evaluation
  • Multimodal interaction

Cite this