Improving what cross-modal retrieval models learn through object-oriented inter- and intra-modal attention networks

Po-Yao Huang, Vaibhav, Xiaojun Chang, Alexander G. Hauptmann

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)


Although significant progress has been made for cross-modal retrieval models in recent years, few have explored what those models truly learn and what makes one model superior to another. Start by training two state-of-the-art text-to-image retrieval models with adversarial text inputs, we investigate and quantify the importance of syntactic structure and lexical information in learning the joint visual-semantic embedding space for cross-modal retrieval. The results show that the retrieval power mainly comes from localizing and connecting the visual objects and their cross-modal counterparts, the textual phrases. Inspired by this observation, we propose a novel model which employs object-oriented encoders along with inter- and intra-modal attention networks to improve inter-modal dependencies for cross-modal retrieval. In addition, we develop a new multimodal structure-preserving objective which additionally emphasizes intra-modal hard negative examples to promote intra-modal discrepancies. Extensive experiments show that the proposed approach outperforms the existing best method by a large margin (16.4% and 6.7% relatively with Recall@1 in the text-toimage retrieval task on the Flickr30K dataset and the MS-COCO dataset respectively).

Original languageEnglish
Title of host publicationICMR’19 - Proceedings of the 2019 ACM International Conference on Multimedia Retrieval
Subtitle of host publicationJune 10–13, 2019, Ottawa, ON, Canada
EditorsK. Selcuk Candan, Marco Bertini, Lixing Xie, Xiao-Yong Wei
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Number of pages9
ISBN (Electronic)9781450367653
Publication statusPublished - 2019
EventACM International Conference on Multimedia Retrieval 2019 - Ottawa, Canada
Duration: 10 Jun 201913 Jun 2019
Conference number: 9th


ConferenceACM International Conference on Multimedia Retrieval 2019
Abbreviated titleICMR 2019
Internet address


  • Cross modal retrieval
  • Joint embedding
  • Text-image matching

Cite this