Abstract
Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting focusing on these moments as turning points (TPs), accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence highlighting changes in emotions, behaviors, perspectives, and decisions at these turning points. We also propose a framework, TPMaven, utilizing state-of-the-art vision-language models to construct a narrative from the videos and large language models to classify and detect turning points in our multi-modal dataset. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.
Original language | English |
---|---|
Title of host publication | ACL 2024, The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Proceedings of the Conference Volume 2: Short Papers |
Editors | Lun-Wei Ku, Andre F. T. Martins, Vivek Srikumar |
Place of Publication | Kerrville TX USA |
Publisher | Association for Computational Linguistics (ACL) |
Pages | 314-326 |
Number of pages | 13 |
ISBN (Electronic) | 9798891760950 |
DOIs | |
Publication status | Published - 2024 |
Event | Annual Meeting of the Association of Computational Linguistics 2024 - Bangkok, Thailand Duration: 11 Aug 2024 → 16 Aug 2024 Conference number: 62nd https://aclanthology.org/2024.acl-long.0/ (Proceedings) https://2024.aclweb.org/ (Website) https://aclanthology.org/volumes/2024.findings-acl/ (Proceedings (Findings)) https://aclanthology.org/volumes/2024.acl-long/ (Proceedings) |
Publication series
Name | Proceedings of the Annual Meeting of the Association for Computational Linguistics |
---|---|
Publisher | Association for Computational Linguistics (ACL) |
Volume | 2 |
ISSN (Print) | 0736-587X |
Conference
Conference | Annual Meeting of the Association of Computational Linguistics 2024 |
---|---|
Abbreviated title | ACL 2024 |
Country/Territory | Thailand |
City | Bangkok |
Period | 11/08/24 → 16/08/24 |
Internet address |
|