Selective dependency aggregation for action classification

Yi Tan, Yanbin Hao, Xiangnan He, Yinwei Wei, Xun Yang

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearch

21 Citations (Scopus)


Video data are distinct from images for the extra temporal dimension, which results in more content dependencies from various perspectives. It increases the difficulty of learning representation for various video actions. Existing methods mainly focus on the dependency under a specific perspective, which cannot facilitate the categorization of complex video actions. This paper proposes a novel selective dependency aggregation (SDA) module, which adaptively exploits multiple types of video dependencies to refine the features. Specifically, we empirically investigate various long-range and short-range dependencies achieved by the multi-direction multi-scale feature squeeze and the dependency excitation. Query structured attention is then adopted to fuse them selectively, fully considering the diversity of videos' dependency preferences. Moreover, the channel reduction mechanism is involved in SDA for controlling the additional computation cost to be lightweight. Finally, we show that the SDA module can be easily plugged into different backbones to form SDA-Nets and demonstrate its effectiveness, efficiency and robustness by conducting extensive experiments on several video benchmarks for action classification. The code and models will be available at

Original languageEnglish
Title of host publicationProceedings of the 29th ACM International Conference on Multimedia
EditorsLiqiang Nie, Qianru Sun, Peng Cui
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages10
ISBN (Electronic)9781450386517
Publication statusPublished - 2021
Externally publishedYes
EventACM International Conference on Multimedia 2021 - Chengdu, China
Duration: 20 Oct 202124 Oct 2021
Conference number: 29th (Proceedings) (Website)


ConferenceACM International Conference on Multimedia 2021
Abbreviated titleMM 2021
Internet address


  • action classification
  • selective depen-dency aggregation
  • video content dependency

Cite this