Tutoring robots: multiparty multimodal social dialogue with an embodied tutor

Samer Al Moubayed, Jonas Beskow, Bajibabu Bollepalli, Ahmed Hussen-Abdelaziz, Martin Johansson, Maria Koutsombogera, José David Lopes, Jekaterina Novikova, Catharine Oertel, Gabriel Skantze, Kalin Stefanov, Gül Varol

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch


This project explores a novel experimental setup towards building spoken, multi-modally rich, and human-like multiparty tutoring agent. A setup is developed and a corpus is collected that targets the development of a dialogue system platform to explore verbal and nonverbal tutoring strategies in multiparty spoken interactions with embodied agents. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. With the participants sits a tutor that helps the participants perform the task and organizes and balances their interaction. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies were coupled with manual annotations to build a situated model of the interaction based on the participants personalities, their temporally-changing state of attention, their conversational engagement and verbal dominance, and the way these are correlated with the verbal and visual feedback, turn-management, and conversation regulatory actions generated by the tutor. At the end of this chapter we discuss the potential areas of research and developments this work opens and some of the challenges that lie in the road ahead.

Original languageEnglish
Title of host publicationInnovative and Creative Developments in Multimodal Interaction Systems
Subtitle of host publication9th IFIPWG 5.5 International SummerWorkshop on Multimodal Interfaces, eNTERFACE 2013 Lisbon, Portugal, July 15 – August 9, 2013 Proceedings
EditorsYves Rybarczyk, Tiago Cardoso, João Rosas, Luis M. Camarinha-Matos
Place of PublicationBerlin German
Number of pages34
ISBN (Electronic)9783642551437
ISBN (Print)9783642551420
Publication statusPublished - 2013
Externally publishedYes
EventIFIPWG 5.5 International Summer Workshop on Multimodal Interfaces 2013 - Lisbon, Portugal
Duration: 15 Jul 20139 Aug 2013
Conference number: 9th
https://link.springer.com/book/10.1007/978-3-642-55143-7 (Proceedings)

Publication series

NameIFIP Advances in Information and Communication Technology
ISSN (Print)1868-422X
ISSN (Electronic)1868-4238


ConferenceIFIPWG 5.5 International Summer Workshop on Multimodal Interfaces 2013
Abbreviated titleeNTERFACE 2013
Internet address


  • Conversational Dominance
  • Embodied Agent
  • Multimodal
  • Multiparty
  • Non-verbal Signals
  • Social Robot
  • Spoken Dialogue
  • Turn-taking
  • Tutor
  • Visual Attention

Cite this