Memory-based network for scene graph with unbalanced relations

Weitao Wang, Ruyang Liu, Meng Wang, Sen Wang, Xiaojun Chang, Yang Chen

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

The scene graph which can be represented by a set of visual triples is composed of objects and the relations between object pairs. It is vital for image captioning, visual question answering, and many other applications. However, there is a long tail distribution on the scene graph dataset, and the tail relation cannot be accurately identified due to the lack of training samples. The problem of the nonstandard label and feature overlap on the scene graph affects the extraction of discriminative features and exacerbates the effect of data imbalance on the model. For these reasons, we propose a novel scene graph generation model that can effectively improve the detection of low-frequency relations. We use the method of memory features to realize the transfer of high-frequency relation features to low-frequency relation features. Extensive experiments on scene graph datasets show that our model significantly improved the performance of two evaluation metrics R@K and mR@K compared with state-of-the-art baselines.

Original languageEnglish
Title of host publicationProceedings of the 28th ACM International Conference on Multimedia
EditorsGuo-Jun Qi, Elisa Ricci, Zhengyou Zhang , Roger Zimmermann
Place of PublicationNew York NY USA
PublisherAssociation for Computing Machinery (ACM)
Pages2400-2408
Number of pages9
ISBN (Electronic)9781450379885
DOIs
Publication statusPublished - 2020
EventACM International Conference on Multimedia 2020 - Virtual, Online, United States of America
Duration: 12 Oct 202016 Oct 2020
Conference number: 28th
https://dl.acm.org/doi/proceedings/10.1145/3394171 (Proceedings)

Conference

ConferenceACM International Conference on Multimedia 2020
Abbreviated titleMM 2020
CountryUnited States of America
CityVirtual, Online
Period12/10/2016/10/20
Internet address

Keywords

  • memory feature
  • neural networks
  • scene graph

Cite this