Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Ed Kaiser; Alex Olwal; David McGee; Hrvoje Benko; Andrea Corradini; Xiaoguang Li; Phil Cohen; Steven Feiner

Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

Ed Kaiser, Alex Olwal, David McGee, Hrvoje Benko, Andrea Corradini, Xiaoguang Li, Phil Cohen, Steven Feiner

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

Original language	English
Title of host publication	ICMI'03
Subtitle of host publication	Fifth International Conference on Multimodal Interfaces
Pages	12-19
Number of pages	8
Publication status	Published - 1 Dec 2003
Externally published	Yes
Event	International Conference on Multimodal Interfaces 2003 - Vancouver, Canada Duration: 5 Nov 2003 → 7 Nov 2003 Conference number: 5th https://dl.acm.org/doi/proceedings/10.1145/958432 (Proceedings)

Publication series

Name	ICMI'03: Fifth International Conference on Multimodal Interfaces

Conference

Conference	International Conference on Multimodal Interfaces 2003
Abbreviated title	ICMI 2003
Country	Canada
City	Vancouver
Period	5/11/03 → 7/11/03
Internet address	https://dl.acm.org/doi/proceedings/10.1145/958432 (Proceedings)

Keywords

Augmented/virtual reality
Evaluation
Multimodal interaction

Access to Document

Link to publication in Scopus

Cite this

@inproceedings{3cd2620f71284fe6ac5adc202f6c9f6c,

title = "Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality",

abstract = "We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.",

keywords = "Augmented/virtual reality, Evaluation, Multimodal interaction",

author = "Ed Kaiser and Alex Olwal and David McGee and Hrvoje Benko and Andrea Corradini and Xiaoguang Li and Phil Cohen and Steven Feiner",

year = "2003",

month = dec,

day = "1",

language = "English",

isbn = "1581136218",

series = "ICMI'03: Fifth International Conference on Multimodal Interfaces",

pages = "12--19",

booktitle = "ICMI'03",

note = "International Conference on Multimodal Interfaces 2003, ICMI 2003 ; Conference date: 05-11-2003 Through 07-11-2003",

url = "https://dl.acm.org/doi/proceedings/10.1145/958432",

}

Kaiser, E, Olwal, A, McGee, D, Benko, H, Corradini, A, Li, X, Cohen, P & Feiner, S 2003, Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. in ICMI'03: Fifth International Conference on Multimodal Interfaces. ICMI'03: Fifth International Conference on Multimodal Interfaces, pp. 12-19, International Conference on Multimodal Interfaces 2003, Vancouver, Canada, 5/11/03.

Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. / Kaiser, Ed; Olwal, Alex; McGee, David; Benko, Hrvoje; Corradini, Andrea; Li, Xiaoguang; Cohen, Phil; Feiner, Steven.

ICMI'03: Fifth International Conference on Multimodal Interfaces. 2003. p. 12-19 (ICMI'03: Fifth International Conference on Multimodal Interfaces).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality

AU - Kaiser, Ed

AU - Olwal, Alex

AU - McGee, David

AU - Benko, Hrvoje

AU - Corradini, Andrea

AU - Li, Xiaoguang

AU - Cohen, Phil

AU - Feiner, Steven

N1 - Conference code: 5th

PY - 2003/12/1

Y1 - 2003/12/1

N2 - We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

AB - We describe an approach to 3D multimodal interaction in immersive augmented and virtual reality environments that accounts for the uncertain nature of the information sources. The resulting multimodal system fuses symbolic and statistical information from a set of 3D gesture, spoken language, and referential agents. The referential agents employ visible or invisible volumes that can be attached to 3D trackers in the environment, and which use a time-stamped history of the objects that intersect them to derive statistics for ranking potential referents. We discuss the means by which the system supports mutual disambiguation of these modalities and information sources, and show through a user study how mutual disambiguation accounts for over 45% of the successful 3D multimodal interpretations. An accompanying video demonstrates the system in action.

KW - Augmented/virtual reality

KW - Evaluation

KW - Multimodal interaction

UR - http://www.scopus.com/inward/record.url?scp=11144225561&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:11144225561

SN - 1581136218

T3 - ICMI'03: Fifth International Conference on Multimodal Interfaces

SP - 12

EP - 19

BT - ICMI'03

T2 - International Conference on Multimodal Interfaces 2003

Y2 - 5 November 2003 through 7 November 2003

ER -