MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations

Gia-Bao Dinh Ho, Chang Wei Tan, Zahra Zamanzadeh Darban, Mahsa Salehi, Gholamreza Haffari, Wray Buntine

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence highlighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.

Original languageEnglish
Title of host publicationACL 2024, The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Proceedings of the Conference Volume 2: Short Papers
EditorsLun-Wei Ku, Andre F. T. Martins, Vivek Srikumar
Place of PublicationKerrville TX USA
PublisherAssociation for Computational Linguistics (ACL)
Pages314-326
Number of pages13
ISBN (Electronic)9798891760950
DOIs
Publication statusPublished - 2024
EventAnnual Meeting of the Association of Computational Linguistics 2024 - Bangkok, Thailand
Duration: 11 Aug 202416 Aug 2024
Conference number: 62nd
https://aclanthology.org/2024.acl-long.0/ (Proceedings)
https://2024.aclweb.org/ (Website)
https://aclanthology.org/volumes/2024.findings-acl/ (Proceedings (Findings))
https://aclanthology.org/volumes/2024.acl-long/ (Proceedings)

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
PublisherAssociation for Computational Linguistics (ACL)
Volume2
ISSN (Print)0736-587X

Conference

ConferenceAnnual Meeting of the Association of Computational Linguistics 2024
Abbreviated titleACL 2024
Country/TerritoryThailand
CityBangkok
Period11/08/2416/08/24
Internet address

Cite this