The role of natural language in a multimodal interface

Philip R. Cohen

doi:10.1145/142621.142641

The role of natural language in a multimodal interface

Philip R. Cohen

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

87 Citations (Scopus)

Abstract

Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language p roblems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

Original language	English
Title of host publication	Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992
Publisher	Association for Computing Machinery (ACM)
Pages	143-149
Number of pages	7
ISBN (Electronic)	0897915496, 9780897915496
DOIs	https://doi.org/10.1145/142621.142641
Publication status	Published - 1 Dec 1992
Externally published	Yes
Event	ACM Symposium on User Interface Software and Technology 1992 - Monteray, United States of America Duration: 15 Nov 1992 → 18 Nov 1992 Conference number: 5th https://dl.acm.org/doi/proceedings/10.1145/142621

Publication series

Name	Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992

Conference

Conference	ACM Symposium on User Interface Software and Technology 1992
Abbreviated title	UIST 1992
Country/Territory	United States of America
City	Monteray
Period	15/11/92 → 18/11/92
Internet address	https://dl.acm.org/doi/proceedings/10.1145/142621

Access to Document

10.1145/142621.142641

Cite this

Cohen, P. R. (1992). The role of natural language in a multimodal interface. In Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992 (pp. 143-149). (Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992). Association for Computing Machinery (ACM). https://doi.org/10.1145/142621.142641

@inproceedings{f679d010575a42da96c536b97ca6f5f3,

title = "The role of natural language in a multimodal interface",

abstract = "Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language p roblems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.",

author = "Cohen, {Philip R.}",

year = "1992",

month = dec,

day = "1",

doi = "10.1145/142621.142641",

language = "English",

series = "Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992",

publisher = "Association for Computing Machinery (ACM)",

pages = "143--149",

booktitle = "Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992",

address = "United States of America",

note = "ACM Symposium on User Interface Software and Technology 1992, UIST 1992 ; Conference date: 15-11-1992 Through 18-11-1992",

url = "https://dl.acm.org/doi/proceedings/10.1145/142621",

}

Cohen, PR 1992, The role of natural language in a multimodal interface. in Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992. Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992, Association for Computing Machinery (ACM), pp. 143-149, ACM Symposium on User Interface Software and Technology 1992, Monteray, California, United States of America, 15/11/92. https://doi.org/10.1145/142621.142641

The role of natural language in a multimodal interface. / Cohen, Philip R.
Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992. Association for Computing Machinery (ACM), 1992. p. 143-149 (Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - The role of natural language in a multimodal interface

AU - Cohen, Philip R.

N1 - Conference code: 5th

PY - 1992/12/1

Y1 - 1992/12/1

N2 - Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language p roblems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

AB - Although graphics and direct manipulation are effective interface technologies for some classes of problems, they are limited in many ways. In particular, they provide little support for identifying objects not on the screen, for specifying temporal relations, for identifying and operating on large sets and subsets of entities, and for using the context of interaction. On the other hand, these are precisely strengths of natural language. This paper presents an interface that blends natural language processing and direct manipulation technologies, using each for their characteristic advantages. Specifically, the paper shows how to use natural language to describe objects and temporal relations, and how to use direct manipulation for overcoming hard natural language p roblems involving the establishment and use of context and pronominal reference. This work has been implemented in SRI's Shoptalk system, a prototype information and decision-support system for manufacturing.

UR - http://www.scopus.com/inward/record.url?scp=84880893970&partnerID=8YFLogxK

U2 - 10.1145/142621.142641

DO - 10.1145/142621.142641

M3 - Conference Paper

AN - SCOPUS:84880893970

T3 - Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992

SP - 143

EP - 149

BT - Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST 1992

PB - Association for Computing Machinery (ACM)

T2 - ACM Symposium on User Interface Software and Technology 1992

Y2 - 15 November 1992 through 18 November 1992

ER -

The role of natural language in a multimodal interface

Abstract

Publication series

Conference

Access to Document

Other files and links

Cite this