Pluralistic image completion

Chuanxia Zheng, Tat Jen Cham, Jianfei Cai

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

449 Citations (Scopus)

Abstract

Most image completion methods produce only one result for each masked input, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion - the task of generating multiple and diverse plausible solutions for image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label. As such, sampling from conditional VAEs still leads to minimal diversity. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one given ground truth to get prior distribution of missing parts and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by GANs. We also introduce a new short+long term attention layer that exploits distant relations among decoder and encoder features, improving appearance consistency. When tested on datasets with buildings (Paris), faces (CelebA-HQ), and natural images (ImageNet), our method not only generated higherquality completion results, but also with multiple and diverse plausible outputs.
Original languageEnglish
Title of host publicationProceedings - 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019
EditorsAbhinav Gupta, Derek Hoiem, Gang Hua, Zhuowen Tu
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages1438-1447
Number of pages10
ISBN (Electronic)9781728132938
ISBN (Print)9781728132945
DOIs
Publication statusPublished - 2019
Externally publishedYes
EventIEEE Conference on Computer Vision and Pattern Recognition 2019 - Long Beach, United States of America
Duration: 16 Jun 201920 Jun 2019
Conference number: 32nd
http://cvpr2019.thecvf.com/
https://ieeexplore.ieee.org/xpl/conhome/8938205/proceeding (Proceedings)

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume2019-June
ISSN (Print)1063-6919

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2019
Abbreviated titleCVPR 2019
Country/TerritoryUnited States of America
CityLong Beach
Period16/06/1920/06/19
Internet address

Keywords

  • Deep Learning
  • Image and Video Synthesis

Cite this