Self-supervised relationship probing

Jiuxiang Gu, Jason Kuen, Shafiq Joty, Jianfei Cai, Vlad I. Morariu, Handong Zhao, Tong Sun

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

1 Citation (Scopus)


Structured representations of images that model visual relationships are beneficial for many vision and vision-language applications. However, current human-annotated visual relationship datasets suffer from the long-tailed predicate distribution problem which limits the potential of visual relationship models. In this work, we introduce a self-supervised method that implicitly learns the visual relationships without relying on any ground-truth visual relationship annotations. Our method relies on 1) intra- and inter-modality encodings to respectively model relationships within each modality separately and jointly, and 2) relationship probing, which seeks to discover the graph structure within each modality. By leveraging masked language modeling, contrastive learning, and dependency tree distances for self-supervision, our method learns better object features as well as implicit visual relationships. We verify the effectiveness of our proposed method on various vision-language tasks that benefit from improved visual relationship understanding.

Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 33 (NeurIPS 2020)
EditorsH. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin
Place of PublicationSan Diego CA USA
PublisherNeural Information Processing Systems (NIPS)
Number of pages13
Publication statusPublished - 2020
EventAdvances of Neural Information Processing Systems 2020 - Online, Virtual, Online, United States of America
Duration: 6 Dec 202012 Dec 2020
Conference number: 34th (Proceedings ) (Website)

Publication series

NameAdvances in Neural Information Processing Systems
PublisherMorgan Kaufmann Publishers
ISSN (Print)1049-5258


ConferenceAdvances of Neural Information Processing Systems 2020
Abbreviated titleNeurIPS 2020
Country/TerritoryUnited States of America
CityVirtual, Online
Internet address

Cite this