DeSIQ: Towards an unbiased, challenging benchmark for Social Intelligence understanding

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. One representative benchmark for its study is Social Intelligence Queries (Social-IQ), a dataset of multiple-choice questions on videos of complex social interactions. We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. Our analysis reveals that Social-IQ contains substantial biases, which can be exploited by a moderately strong language model to learn spurious correlations to achieve perfect performance without being given the context or even the question. We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ. Our empirical analysis shows DeSIQ significantly reduces the biases in the original Social-IQ dataset. Furthermore, we examine and shed light on the effect of model size, model style, learning settings, commonsense knowledge, and multi-modality on the new benchmark performance. Our new dataset, observations and findings open up important research questions for the study of social intelligence.

Original languageEnglish
Title of host publicationProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
EditorsNadi Tomeh, Atsushi Fujita, Aixin Sun, Bin Wang, Rong Tong
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Pages3169-3180
Number of pages12
ISBN (Electronic)9798891760608
DOIs
Publication statusPublished - 2023
EventEmpirical Methods in Natural Language Processing 2023 - , Singapore
Duration: 6 Dec 202310 Dec 2023
https://2023.emnlp.org/
https://aclanthology.org/volumes/2023.findings-emnlp/ (Proceedings)
https://aclanthology.org/volumes/2023.emnlp-demo/ (Proceedings)

Conference

ConferenceEmpirical Methods in Natural Language Processing 2023
Abbreviated titleEMNLP 2023
Country/TerritorySingapore
Period6/12/2310/12/23
Internet address

Cite this