Creating tangible interfaces by augmenting physical objects with multimodal language

D. R. McGee, P. R. Cohen

Research output: Contribution to conferencePaperpeer-review

33 Citations (Scopus)


Rasa is a tangible augmented reality environment that digitally enhances the existing paper-based command and control capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa computationally augments the physical objects on a command post map, linking these items to digital representations of the same - for example, linking a paper map to the world and Post-it™ notes to military units. Herein, we give a thorough account of Rasa's underlying multiagent framework, and its recognition, understanding, and multimodal integration components. Moreover, we examine five properties of language - generativity, comprehensibility, compositionality, referentiality, and, at times, persistence - that render it suitable as an augmentation approach, contrasting these properties to those of other augmentation methods. It is these properties of language that allow users of Rasa to augment physical objects, transforming them into tangible interfaces.

Original languageEnglish
Number of pages7
Publication statusPublished - 1 Jan 2001
Externally publishedYes
Event2001 International Conference on Intelligent User Interfaces (IUI 2001) - Santa Fe, NM, United States of America
Duration: 14 Jan 200117 Jan 2001


Conference2001 International Conference on Intelligent User Interfaces (IUI 2001)
Country/TerritoryUnited States of America
CitySanta Fe, NM


  • Augmented reality
  • Human factors
  • Mixed reality
  • Multimodal interfaces
  • Tangible interfaces, and invisible interfaces

Cite this