CA-OVS: Cluster and Adapt Mask Proposals for Open-Vocabulary Semantic Segmentation

Recent Open-Vocabulary Semantic Segmentation (OVS) works typically follow the mask proposal pipeline that decouples semantic segmentation into class-agnostic mask generation and mask-class matching. They train mask generation modules on segmentation datasets, while learning mask-class matching from pretrained vision-language models and large-scale image classification datasets to recognize open-vocabulary classes. There are two major challenges in this pipeline during training: 1) mismatching between mask proposals and classes, as well as 2) domain and label gaps between classification and segmentation datasets. In this paper, we propose a novel CA-OVS framework to solve these challenges. For the first challenge, a Wasserstein-distance-based clustering method is presented to better match masks and classes. For the second challenge, we propose to transfer the information of the mask proposals from the segmentation dataset to the classification dataset by minimizing their Wasserstein distance. Extensive experiments on several OVS datasets show that our method outperforms many state-of-the-art approaches.