Error resolution during multimodal human-computer interaction

Recent research indicates clear performance advantages and a strong user preference for interacting multimodally with computers. However, in the problematic area of error resolution, possible advantages of multimodal interface design remain poorly understood. In the present research, a semi-automatic simulation method with a novel error-generation capability was used to collect within-subject data before and after recognition errors, and at different spiral depths in terms of number of repetitions required to resolve an error. Results indicated that users adopt a strategy of switching input modalities and lexical expressions when resolving errors, strategies that they use in a linguistically contrastive manner to distinguish a repetition from original failed input. Implications of these findings are discussed for the development of user-centered predictive models of linguistic adaptation during human-computer error resolution, and for the development of improved error handling in advanced recognition-based interfaces.