ViLPAct: A benchmark for compositional generalization on multimodal human activities

Terry Yue Zhuo; Yaqing Liao; Yuecheng Lei; Lizhen Qu; Gerard de Melo; Xiaojun Chang; Yazhou Ren; Zenglin Xu

ViLPAct: A benchmark for compositional generalization on multimodal human activities

Terry Yue Zhuo, Yaqing Liao, Yuecheng Lei, Lizhen Qu, Gerard de Melo, Xiaojun Chang, Yazhou Ren, Zenglin Xu

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from Charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods.

Original language	English
Title of host publication	EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023
Editors	Rya Cotterell, Carolina Scarton
Place of Publication	Stroudsburg PA USA
Publisher	Association for Computational Linguistics (ACL)
Pages	2192-2207
Number of pages	16
ISBN (Electronic)	9781959429470
Publication status	Published - 2023
Event	European Association of Computational Linguistics Conference 2023 - Dubrovnik, Croatia Duration: 2 May 2023 → 6 May 2023 Conference number: 17th https://2023.eacl.org/ (Website) https://aclanthology.org/volumes/2023.eacl-main/ (Proceedings)

Conference

Conference	European Association of Computational Linguistics Conference 2023
Abbreviated title	EACL 2023
Country/Territory	Croatia
City	Dubrovnik
Period	2/05/23 → 6/05/23
Internet address	https://2023.eacl.org/ (Website) https://aclanthology.org/volumes/2023.eacl-main/ (Proceedings)

Access to Document

501592532-oaFinal published version, 997 KB

https://aclanthology.org/2023.findings-eacl.164

Cite this

Zhuo, T. Y., Liao, Y., Lei, Y., Qu, L., de Melo, G., Chang, X., Ren, Y., & Xu, Z. (2023). ViLPAct: A benchmark for compositional generalization on multimodal human activities. In R. Cotterell, & C. Scarton (Eds.), EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023 (pp. 2192-2207). Association for Computational Linguistics (ACL). https://aclanthology.org/2023.findings-eacl.164

Zhuo, Terry Yue ; Liao, Yaqing ; Lei, Yuecheng et al. / ViLPAct : A benchmark for compositional generalization on multimodal human activities. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023. editor / Rya Cotterell ; Carolina Scarton. Stroudsburg PA USA : Association for Computational Linguistics (ACL), 2023. pp. 2192-2207

@inproceedings{45e9eb45595d438cbc69983d1e989ee5,

title = "ViLPAct: A benchmark for compositional generalization on multimodal human activities",

abstract = "We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from Charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods.",

author = "Zhuo, {Terry Yue} and Yaqing Liao and Yuecheng Lei and Lizhen Qu and {de Melo}, Gerard and Xiaojun Chang and Yazhou Ren and Zenglin Xu",

note = "Funding Information: This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), the Major Key Project of PCL (No. PCL2021A06), an Open Research Project of Zhejiang Lab (NO.2022RC0AB04), and Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (No. 2022B1212010005). Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; European Association of Computational Linguistics Conference 2023, EACL 2023 ; Conference date: 02-05-2023 Through 06-05-2023",

year = "2023",

language = "English",

pages = "2192--2207",

editor = "Rya Cotterell and Carolina Scarton",

booktitle = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023",

publisher = "Association for Computational Linguistics (ACL)",

url = "https://2023.eacl.org/, https://aclanthology.org/volumes/2023.eacl-main/",

}

Zhuo, TY, Liao, Y, Lei, Y, Qu, L, de Melo, G, Chang, X, Ren, Y & Xu, Z 2023, ViLPAct: A benchmark for compositional generalization on multimodal human activities. in R Cotterell & C Scarton (eds), EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023. Association for Computational Linguistics (ACL), Stroudsburg PA USA, pp. 2192-2207, European Association of Computational Linguistics Conference 2023, Dubrovnik, Croatia, 2/05/23. <https://aclanthology.org/2023.findings-eacl.164>

ViLPAct: A benchmark for compositional generalization on multimodal human activities. / Zhuo, Terry Yue; Liao, Yaqing; Lei, Yuecheng et al.
EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023. ed. / Rya Cotterell; Carolina Scarton. Stroudsburg PA USA: Association for Computational Linguistics (ACL), 2023. p. 2192-2207.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - ViLPAct

T2 - European Association of Computational Linguistics Conference 2023

AU - Zhuo, Terry Yue

AU - Liao, Yaqing

AU - Lei, Yuecheng

AU - Qu, Lizhen

AU - de Melo, Gerard

AU - Chang, Xiaojun

AU - Ren, Yazhou

AU - Xu, Zenglin

N1 - Conference code: 17th

PY - 2023

Y1 - 2023

N2 - We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from Charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods.

AB - We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from Charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods.

UR - http://www.scopus.com/inward/record.url?scp=85159854855&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:85159854855

SP - 2192

EP - 2207

BT - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023

A2 - Cotterell, Rya

A2 - Scarton, Carolina

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg PA USA

Y2 - 2 May 2023 through 6 May 2023

ER -

Zhuo TY, Liao Y, Lei Y, Qu L, de Melo G, Chang X et al. ViLPAct: A benchmark for compositional generalization on multimodal human activities. In Cotterell R, Scarton C, editors, EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023. Stroudsburg PA USA: Association for Computational Linguistics (ACL). 2023. p. 2192-2207

ViLPAct: A benchmark for compositional generalization on multimodal human activities

Abstract

Conference

Access to Document

Other files and links

Cite this