Abstract
As a new generation of multimodal systems begins to emerge, one dominant theme will be the integration and synchronization requirements for combining modalities into robust whole systems. In the present research, quantitative modeling is presented on the organization of users' speech and pen multimodal integration patterns. In particular, the potential malleability of users' multimodal integration patterns is explored, as well as variation in these patterns during system error handling and tasks varying in difficulty. Using a new dual-wizard simulation method, data was collected from twelve adults as they interacted with a map-based task using multimodal speech and pen input. Analyses based on over 1600 multimodal constructions revealed that users' dominant multimodal integration pattern was resistant to change, even when strong selective reinforcement was delivered to encourage switching from a sequential to simultaneous integration pattern, or vice versa. Instead, both sequential and simultaneous integrators showed evidence of entrenching further in their dominant integration patterns (i.e., increasing either their inter-modal lag or signal overlap) over the course of an interactive session, during system error handling, and when completing increasingly difficult tasks. In fact, during error handling these changes in the co-timing of multimodal signals became the main feature of hyper-clear multimodal language, with elongation of individual signals either attenuated or absent. Whereas Behavioral/Structuralist theory cannot account for these data, it is argued that Gestalt theory provides a valuable framework and insights into multimodal interaction. Implications of these findings are discussed for the development of a coherent theory of multimodal integration during human-computer interaction, and for the design of a new class of adaptive multimodal interfaces.
Original language | English |
---|---|
Title of host publication | ICMI'03 |
Subtitle of host publication | Fifth International Conference on Multimodal Interfaces |
Place of Publication | New York NY USA |
Publisher | Association for Computing Machinery (ACM) |
Pages | 44-51 |
Number of pages | 8 |
ISBN (Print) | 1581136218 |
Publication status | Published - 2003 |
Externally published | Yes |
Event | International Conference on Multimodal Interfaces 2003 - Vancouver, Canada Duration: 5 Nov 2003 → 7 Nov 2003 Conference number: 5th https://dl.acm.org/doi/proceedings/10.1145/958432 (Proceedings) |
Conference
Conference | International Conference on Multimodal Interfaces 2003 |
---|---|
Abbreviated title | ICMI 2003 |
Country/Territory | Canada |
City | Vancouver |
Period | 5/11/03 → 7/11/03 |
Internet address |
|
Keywords
- Co-timing
- Entrenchment
- Error handling
- Gestalt theory
- Multimodal integration
- Speech and pen input
- Task difficulty