Look who's talking: visual identification of the active speaker in multi-party human-robot interaction

Kalin Stefanov; Akihiro Sugimoto; Jonas Beskow

doi:10.1145/3005467.3005470

Look who's talking: visual identification of the active speaker in multi-party human-robot interaction

Kalin Stefanov, Akihiro Sugimoto, Jonas Beskow

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

4 Citations (Scopus)

Abstract

This paper presents analysis of a previously recorded multimodal interaction dataset. The primary purpose of that dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper presents a data-driven methodology for automatic visual identification of the active speaker based on facial action units (AUs). The paper also presents an evaluation of the proposed methodology on 12 different interactions with an approximate length of 4 hours. The methodology will be implemented on a robot and used to generate natural focus of visual attention behavior during multi-party human-robot interactions.

Original language	English
Title of host publication	2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016)
Editors	Louis-Philippe Morency, Carlos Busso, Catherine Pelachaud
Place of Publication	New York NY USA
Publisher	Association for Computing Machinery (ACM)
Pages	22-27
Number of pages	6
ISBN (Electronic)	9781450345576
DOIs	https://doi.org/10.1145/3005467.3005470
Publication status	Published - 2016
Externally published	Yes
Event	Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 - Tokyo, Japan Duration: 16 Nov 2016 → 16 Nov 2016 Conference number: 2nd https://dl.acm.org/doi/proceedings/10.1145/3005467 (Proceedings) https://web.archive.org/web/20160804170758/https://wwwhome.ewi.utwente.nl/~truongkp/icmi2016-assp4mi (Website)

Conference

Conference	Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016
Abbreviated title	ASSP4MI 2016
Country	Japan
City	Tokyo
Period	16/11/16 → 16/11/16
Internet address	https://dl.acm.org/doi/proceedings/10.1145/3005467 (Proceedings) https://web.archive.org/web/20160804170758/https://wwwhome.ewi.utwente.nl/~truongkp/icmi2016-assp4mi (Website)

Keywords

Active speaker identification
Human-robot interaction
Multi-modal interaction

Access to Document

10.1145/3005467.3005470

Link to publication in Scopus

Cite this

Stefanov, K., Sugimoto, A., & Beskow, J. (2016). Look who's talking: visual identification of the active speaker in multi-party human-robot interaction. In L-P. Morency, C. Busso, & C. Pelachaud (Eds.), 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016) (pp. 22-27). Association for Computing Machinery (ACM). https://doi.org/10.1145/3005467.3005470

Stefanov, Kalin ; Sugimoto, Akihiro ; Beskow, Jonas. / Look who's talking : visual identification of the active speaker in multi-party human-robot interaction. 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016). editor / Louis-Philippe Morency ; Carlos Busso ; Catherine Pelachaud. New York NY USA : Association for Computing Machinery (ACM), 2016. pp. 22-27

@inproceedings{dbaf8c7416d84925a12832c7827eaac3,

title = "Look who's talking: visual identification of the active speaker in multi-party human-robot interaction",

abstract = "This paper presents analysis of a previously recorded multimodal interaction dataset. The primary purpose of that dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper presents a data-driven methodology for automatic visual identification of the active speaker based on facial action units (AUs). The paper also presents an evaluation of the proposed methodology on 12 different interactions with an approximate length of 4 hours. The methodology will be implemented on a robot and used to generate natural focus of visual attention behavior during multi-party human-robot interactions.",

keywords = "Active speaker identification, Human-robot interaction, Multi-modal interaction",

author = "Kalin Stefanov and Akihiro Sugimoto and Jonas Beskow",

year = "2016",

doi = "10.1145/3005467.3005470",

language = "English",

pages = "22--27",

editor = "Morency, {Louis-Philippe } and Busso, {Carlos } and Pelachaud, {Catherine }",

booktitle = "2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016)",

publisher = "Association for Computing Machinery (ACM)",

address = "United States of America",

note = "Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016, ASSP4MI 2016 ; Conference date: 16-11-2016 Through 16-11-2016",

url = "https://dl.acm.org/doi/proceedings/10.1145/3005467, https://web.archive.org/web/20160804170758/https://wwwhome.ewi.utwente.nl/~truongkp/icmi2016-assp4mi",

}

Stefanov, K, Sugimoto, A & Beskow, J 2016, Look who's talking: visual identification of the active speaker in multi-party human-robot interaction. in L-P Morency, C Busso & C Pelachaud (eds), 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016). Association for Computing Machinery (ACM), New York NY USA, pp. 22-27, Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016, Tokyo, Japan, 16/11/16. https://doi.org/10.1145/3005467.3005470

Look who's talking : visual identification of the active speaker in multi-party human-robot interaction. / Stefanov, Kalin; Sugimoto, Akihiro; Beskow, Jonas.

2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016). ed. / Louis-Philippe Morency; Carlos Busso; Catherine Pelachaud. New York NY USA : Association for Computing Machinery (ACM), 2016. p. 22-27.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Look who's talking

T2 - Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016

AU - Stefanov, Kalin

AU - Sugimoto, Akihiro

AU - Beskow, Jonas

N1 - Conference code: 2nd

PY - 2016

Y1 - 2016

N2 - This paper presents analysis of a previously recorded multimodal interaction dataset. The primary purpose of that dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper presents a data-driven methodology for automatic visual identification of the active speaker based on facial action units (AUs). The paper also presents an evaluation of the proposed methodology on 12 different interactions with an approximate length of 4 hours. The methodology will be implemented on a robot and used to generate natural focus of visual attention behavior during multi-party human-robot interactions.

AB - This paper presents analysis of a previously recorded multimodal interaction dataset. The primary purpose of that dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper presents a data-driven methodology for automatic visual identification of the active speaker based on facial action units (AUs). The paper also presents an evaluation of the proposed methodology on 12 different interactions with an approximate length of 4 hours. The methodology will be implemented on a robot and used to generate natural focus of visual attention behavior during multi-party human-robot interactions.

KW - Active speaker identification

KW - Human-robot interaction

KW - Multi-modal interaction

UR - http://www.scopus.com/inward/record.url?scp=85006754103&partnerID=8YFLogxK

U2 - 10.1145/3005467.3005470

DO - 10.1145/3005467.3005470

M3 - Conference Paper

AN - SCOPUS:85006754103

SP - 22

EP - 27

BT - 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016)

A2 - Morency, Louis-Philippe

A2 - Busso, Carlos

A2 - Pelachaud, Catherine

PB - Association for Computing Machinery (ACM)

CY - New York NY USA

Y2 - 16 November 2016 through 16 November 2016

ER -

Stefanov K, Sugimoto A, Beskow J. Look who's talking: visual identification of the active speaker in multi-party human-robot interaction. In Morency L-P, Busso C, Pelachaud C, editors, 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction 2016 (ASSP4MI2016). New York NY USA: Association for Computing Machinery (ACM). 2016. p. 22-27 https://doi.org/10.1145/3005467.3005470