Deep music retrieval for fine-grained videos by exploiting cross-modal-encoded voice-overs

Tingtian Li, Zixun Sun, Haoruo Zhang, Jin Li, Ziming Wu, Hui Zhan, Yipeng Yu, Hengcan Shi

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


Recently, the witness of the rapidly growing popularity of short videos on different Internet platforms has intensified the need for a background music (BGM) retrieval system. However, existing video-music retrieval methods only based on the visual modality cannot show promising performance regarding videos with fine-grained virtual contents. In this paper, we also investigate the widely added voice-overs in short videos and propose a novel framework to retrieve BGM for fine-grained short videos. In our framework, we use the self-attention (SA) and the cross-modal attention (CMA) modules to explore the intra- and the inter-relationships of different modalities respectively. For balancing the modalities, we dynamically assign different weights to the modal features via a fusion gate. For paring the query and the BGM embeddings, we introduce a triplet pseudo-label loss to constrain the semantics of the modal embeddings. As there are no existing virtual-content video-BGM retrieval datasets, we build and release two virtual-content video datasets HoK400 and CFM400. Experimental results show that our method achieves superior performance and outperforms other state-of-the-art methods with large margins.
Original languageEnglish
Title of host publicationProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
EditorsPablo Castells, Rosie Jones, Tetsuya Sakai
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages5
ISBN (Electronic)9781450380379
Publication statusPublished - 2021
EventACM SIGIR Conference on Research and Development in Information Retrieval 2021 - Online, Canada
Duration: 11 Jul 202115 Jul 2021
Conference number: 44th


ConferenceACM SIGIR Conference on Research and Development in Information Retrieval 2021
Abbreviated titleSIGIR 2021

Cite this