Grounding visual concepts for zero-shot Event Detection and Event Captioning

Zhihui Li, Xiaojun Chang, Lina Yao, Shirui Pan, Ge Zongyuan, Huaxiang Zhang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

The flourishing of social media platforms requires techniques for understanding the content of media on a large scale. However, state-of-the art video event understanding approaches remain very limited in terms of their ability to deal with data sparsity, semantically unrepresentative event names, and lack of coherence between visual and textual concepts. Accordingly, in this paper, we propose a method of grounding visual concepts for large-scale Multimedia Event Detection (MED) and Multimedia Event Captioning (MEC) in zero-shot setting. More specifically, our framework composes the following: (1) deriving the novel semantic representations of events from their textual descriptions, rather than event names; (2) aggregating the ranks of grounded concepts for MED tasks. A statistical mean-shift outlier rejection model is proposed to remove the outlying concepts which are incorrectly grounded; and (3) defining MEC tasks and augmenting the MEC training set by the videos detected in MED in a zero-shot setting. To the best of our knowledge, this work is the first time to define and solve the MEC task, which is a further step towards understanding video events. We conduct extensive experiments and achieve state-of-the-art performance on the TRECVID MEDTest dataset, as well as our newly proposed TRECVID-MEC dataset.
Original languageEnglish
Title of host publicationProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
EditorsJiliang Tang, B. Aditya Prakash
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages297–305
Number of pages9
ISBN (Electronic)9781450379984
DOIs
Publication statusPublished - Aug 2020
EventACM International Conference on Knowledge Discovery and Data Mining 2020 - Virtual, Online, United States of America
Duration: 23 Aug 202027 Aug 2020
Conference number: 26th
https://dl.acm.org/doi/proceedings/10.1145/3394486 (Proceedings)
https://www.kdd.org/kdd2020/ (Website)

Conference

ConferenceACM International Conference on Knowledge Discovery and Data Mining 2020
Abbreviated titleKDD 2020
Country/TerritoryUnited States of America
CityVirtual, Online
Period23/08/2027/08/20
Internet address

Keywords

  • Multimedia Event Detection
  • Multimedia Event Captioning
  • Grounding Visual Concepts
  • Zero-shot Learning

Cite this