Human-robot collaborative tutoring using multiparty multimodal spoken dialogue

Samer Al Moubayed, Jonas Beskow, Bajibabu Bollepalli, Joakim Gustafson, Ahmed Hussen-Abdelaziz, Martin Johansson, Maria Koutsombogera, José David Lopes, Jekaterina Novikova, Catharine Oertel, Gabriel Skantze, Kalin Stefanov, Gül Varol

Research output: Chapter in Book/Report/Conference proceedingConference PaperOtherpeer-review

3 Citations (Scopus)

Abstract

In this paper, we describe a project that explores a novel experi-mental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is collect-ed. The corpus targets the development of a dialogue system platform to study verbal and nonverbal tutoring strategies in mul-tiparty spoken interactions with robots which are capable of spo-ken dialogue. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. Along with the participants sits a tutor (robot) that helps the par-ticipants perform the task, and organizes and balances their inter-action. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies, such as a microphone array, Kinects, and video cameras, were coupled with manual annotations. These are used build a situated model of the interaction based on the participants personalities, their state of attention, their conversational engagement and verbal domi-nance, and how that is correlated with the verbal and visual feed-back, turn-management, and conversation regulatory actions gen-erated by the tutor. Driven by the analysis of the corpus, we will show also the detailed design methodologies for an affective, and multimodally rich dialogue system that allows the robot to meas-ure incrementally the attention states, and the dominance for each participant, allowing the robot head Furhat to maintain a well-coordinated, balanced, and engaging conversation, that attempts to maximize the agreement and the contribution to solve the task. This project sets the first steps to explore the potential of us-ing multimodal dialogue systems to build interactive robots that can serve in educational, team building, and collaborative task solving applications.

Original languageEnglish
Title of host publicationProceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction
EditorsTony Belpaeme, Andrea Thomaz
Place of PublicationNew York NY USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages112-113
Number of pages2
ISBN (Electronic)9781450326582
DOIs
Publication statusPublished - 2014
Externally publishedYes
EventAnnual ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2014 - Bielefeld, Germany
Duration: 3 Mar 20146 Mar 2014
Conference number: 9th
https://dl.acm.org/doi/proceedings/10.1145/2559636 (Proceedings)
http://humanrobotinteraction.org/2014/ (Website)

Conference

ConferenceAnnual ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2014
Abbreviated titleHRI 2014
CountryGermany
CityBielefeld
Period3/03/146/03/14
Internet address

Keywords

  • Furhat robot
  • Human-robot collaboration
  • Human-robot interaction
  • Multiparty interaction
  • Spoken dialog

Cite this