Context shifts: Extending the meanings of physical objects with language

The influence that language has on contextual interpretations cannot be ignored by computer systems that strive to be context aware. Rather, once systems are designed to perceive language and other forms of human action, these interpretative processes will of necessity be context dependent. As an example, we illustrate how people simply and naturally create new contexts by naming and referring. We then describe Rasa, a mixed-reality system that observes and understands how users in a military command post create such contexts as part of the process of maintaining situational awareness. In such environments, commander's maps are covered with Post-it® notes. These paper artifacts are contextually transformed to represent units in the field by the application of multimodal language. Rasa understands this language, thereby allowing paper-based tools to become the basis for digital interaction. Finally, we argue that architectures for such context-aware systems will need to be built to process the inherent ambiguity and uncertainty of human input in order to be effective.