The spatially-correlative loss for various image translation tasks

Chuanxia Zheng, Tat Jen Cham, Jianfei Cai

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

75 Citations (Scopus)

Abstract

We propose a novel spatially-correlative loss that is simple, efficient and yet effective for preserving scene structure consistency while supporting large appearance changes during unpaired image-to-image (I2I) translation. Previous methods attempt this by using pixel-level cycle-consistency or feature-level matching losses, but the domain-specific nature of these losses hinder translation across large domain gaps. To address this, we exploit the spatial patterns of self-similarity as a means of defining scene structure. Our spatially-correlative loss is geared towards only capturing spatial relationships within an image rather than domain appearance. We also introduce a new self-supervised learning method to explicitly learn spatially-correlative maps for each specific translation task. We show distinct improvement over baseline models in all three modes of unpaired I2I translation: single-modal, multi-modal, and even single-image translation. This new loss can easily be integrated into existing network architectures and thus allows wide applicability. The code is available at https://github.com/lyndonzheng/F-LSeSim.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021
EditorsMargaux Masson-Forsythe, Eric Mortensen
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages16402-16412
Number of pages11
ISBN (Electronic)9781665445092
ISBN (Print)9781665445108
DOIs
Publication statusPublished - 2021
EventIEEE Conference on Computer Vision and Pattern Recognition 2021 - Online, Virtual, Online, United States of America
Duration: 19 Jun 202125 Jun 2021
https://cvpr2021.thecvf.com/ (Website)
https://ieeexplore.ieee.org/xpl/conhome/9577055/proceeding (Proceedings)

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
PublisherThe Institute of Electrical and Electronics Engineers, Inc.
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075

Conference

ConferenceIEEE Conference on Computer Vision and Pattern Recognition 2021
Abbreviated titleCVPR 2021
Country/TerritoryUnited States of America
CityVirtual, Online
Period19/06/2125/06/21
Internet address

Cite this