State-aware compositional learning towards unbiased training for scene graph generation

Tao He, Lianli Gao, Jingkuan Song, Yuan-Fang Li

Research output: Contribution to journalArticleResearchpeer-review

5 Citations (Scopus)

Abstract

How to avoid biased predictions is an important and active research question in scene graph generation (SGG). Current state-of-the-art methods employ debiasing techniques such as resampling and causality analysis. However, the role of intrinsic cues in the features causing biased training has remained under-explored. In this paper, for the first time, we make the surprising observation that object identity information, in the form of object label embeddings (e.g. GLOVE), is principally responsible for biased predictions. We empirically observe that, even without any visual features, a number of recent SGG models can produce comparable or even better results solely from object label embeddings. Motivated by this insight, we propose to leverage a conditional variational auto-encoder to decouple the entangled visual features into two meaningful components: the object&#x2019;s intrinsic <italic>identity</italic> features and the extrinsic, relation-dependent <italic>state</italic> feature. We further develop two compositional learning strategies on the relation and object levels to mitigate the data scarcity issue of rare relations. On the two benchmark datasets Visual Genome and GQA, we conduct extensive experiments on the three scenarios, i.e., conventional, few-shot and zero-shot SGG. Results consistently demonstrate that our proposed Decomposition and Composition (DeC) method effectively alleviates the biases in the relation prediction. Moreover, DeC is model-free, and it significantly improves the performance of recent SGG models, establishing new state-of-the-art performance.

Original languageEnglish
Pages (from-to)43-56
Number of pages14
JournalIEEE Transactions on Image Processing
Volume32
DOIs
Publication statusPublished - 2 Dec 2022

Keywords

  • Compositional Learning
  • Data Augmentation
  • Feature Decomposition
  • Scene Graph Generation

Cite this