Work together: correlation-identity reconstruction hashing for unsupervised cross-modal retrieval

Lei Zhu, Xize Wu, Jingjing Li, Zheng Zhang, Weili Guan, Heng Tao Shen

Research output: Contribution to journalArticleResearchpeer-review

7 Citations (Scopus)

Abstract

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets at <uri>https://github.com/XizeWu/CIRH</uri>.

Original languageEnglish
Pages (from-to)8838-8851
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume35
Issue number9
DOIs
Publication statusPublished - 1 Sept 2023

Keywords

  • Codes
  • Correlation
  • Correlation-identity
  • Cross-modal retrieval
  • Hash functions
  • Linear matrix inequalities
  • Matrix decomposition
  • Multi-modal correlation
  • Neural networks
  • Reconstruction network
  • Semantic reconstruction
  • Semantics

Cite this