Siamese alignment network for weakly supervised video moment retrieval

Yunxiao Wang, Meng Liu, Yinwei Wei, Zhiyong Cheng, Yinglong Wang, Liqiang Nie

Research output: Contribution to journalArticleResearchpeer-review

6 Citations (Scopus)

Abstract

Video moment retrieval, i.e., localizing the specific video moments within a video given a description query, has attracted substantial attention over the past several years. Although great progress has been achieved thus far, most of existing methods are supervised, which require moment-level temporal annotation information. In contrast, weakly-supervised methods which only need video-level annotations remain largely unexplored. In this paper, we propose a novel end-to-end Siamese alignment network for weakly-supervised video moment retrieval. To be specific, we design a multi-scale Siamese module, which could progressively reduce the semantic gap between the visual and textual modality with the Siamese structure. In addition, we present a context-aware multiple instance learning module by considering the influence of adjacent contexts, enhancing the moment-query and video-query alignment simultaneously. By promoting the matching of both moment-level and video-level, our model can effectively improve the retrieval performance, even if only having weak video level annotations. Extensive experiments on two benchmark datasets, i.e., ActivityNet-Captions and Charades-STA, verify the superiority of our model compared with several state-of-the-art baselines.

Original languageEnglish
Pages (from-to)3921-3933
Number of pages13
JournalIEEE Transactions on Multimedia
Volume25
DOIs
Publication statusPublished - 19 Apr 2022
Externally publishedYes

Keywords

  • Annotations
  • Location awareness
  • Multiple Instance Learning
  • Neural networks
  • Proposals
  • Semantics
  • Siamese Alignment Network
  • Task analysis
  • Vision-Language Alignment
  • Visualization
  • Weakly-supervised Video Moment Retrieval

Cite this