Towards open-vocabulary scene graph generation with prompt-based finetuning

Tao He, Lianli Gao, Jingkuan Song, Yuan-Fang Li

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

7 Citations (Scopus)


Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. The prevailing SGG methods require all object classes to be given in the training set. Such a closed setting limits the practical application of SGG. In this paper, we introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting, in which a model is trained on a set of base object classes but is required to infer relations for unseen target object classes. To this end, we propose a two-step method which firstly pre-trains on large amounts of coarse-grained region-caption data and then leverages two prompt-based techniques to finetune the pre-trained model without updating its parameters. Moreover, our method is able to support inference over completely unseen object classes, which existing methods are incapable of handling. On extensive experiments on three benchmark datasets, Visual Genome, GQA and Open-Image, our method significantly outperforms recent, strong SGG methods on the setting of Ov-SGG, as well as on the conventional closed SGG.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XXVIII
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Place of PublicationCham Switzerland
Number of pages18
ISBN (Electronic)9783031198151
ISBN (Print)9783031198144
Publication statusPublished - 2022
EventEuropean Conference on Computer Vision 2022 - Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022
Conference number: 17th (Proceedings) (Website)

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Computer Vision 2022
Abbreviated titleECCV 2022
CityTel Aviv
Internet address


  • Open-vocabulary scene graph generation
  • Prompt-based finetuning
  • Visual-language model pretraining

Cite this