C4Synth: Cross-caption cycle-consistent text-to-image synthesis

K. J. Joseph; Arghya Pal; Sailaja Rajanala; Vineeth N. Balasubramanian

doi:10.1109/WACV.2019.00044

C4Synth: Cross-caption cycle-consistent text-to-image synthesis

K. J. Joseph, Arghya Pal, Sailaja Rajanala, Vineeth N. Balasubramanian

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

22 Citations (Scopus)

Abstract

Generating an image from its description is a challenging task worth solving because of its numerous practical applications ranging from image editing to virtual reality. All existing methods use one single caption to generate a plausible image. A single caption by itself, can be limited and may not be able to capture the variety of concepts and behavior that would be present in the image. We propose two deep generative models that generate an image by making use of multiple captions describing it. This is achieved by ensuring ‘Cross-Caption Cycle Consistency’ between the multiple captions and the generated image(s). We report quantitative and qualitative results on the standard Caltech-UCSD Birds (CUB) and Oxford-102 Flowers datasets to validate the efficacy of the proposed approach.

Original language	English
Title of host publication	Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018
Editors	Michael Brown, Yanxi Liu, Peyman Milanfar
Place of Publication	Piscataway NJ USA
Publisher	IEEE, Institute of Electrical and Electronics Engineers
Pages	358-366
Number of pages	9
ISBN (Electronic)	9781728119755
ISBN (Print)	9781728119762
DOIs	https://doi.org/10.1109/WACV.2019.00044
Publication status	Published - 2019
Externally published	Yes
Event	IEEE Winter Conference on Applications of Computer Vision 2019 - Waikoloa Village, United States of America Duration: 7 Jan 2019 → 11 Jan 2019 Conference number: 19th https://wacv19.wacv.net/ (Website) https://ieeexplore.ieee.org/xpl/conhome/8642793/proceeding (Proceedings)

Conference

Conference	IEEE Winter Conference on Applications of Computer Vision 2019
Abbreviated title	WACV 2019
Country/Territory	United States of America
City	Waikoloa Village
Period	7/01/19 → 11/01/19
Internet address	https://wacv19.wacv.net/ (Website) https://ieeexplore.ieee.org/xpl/conhome/8642793/proceeding (Proceedings)

Access to Document

10.1109/WACV.2019.00044

Cite this

Joseph, K. J., Pal, A., Rajanala, S., & Balasubramanian, V. N. (2019). C4Synth: Cross-caption cycle-consistent text-to-image synthesis. In M. Brown, Y. Liu, & P. Milanfar (Eds.), Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018 (pp. 358-366). IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/WACV.2019.00044

@inproceedings{d24dfe0337b44029951ad4c6add4fa65,

title = "C4Synth: Cross-caption cycle-consistent text-to-image synthesis",

abstract = "Generating an image from its description is a challenging task worth solving because of its numerous practical applications ranging from image editing to virtual reality. All existing methods use one single caption to generate a plausible image. A single caption by itself, can be limited and may not be able to capture the variety of concepts and behavior that would be present in the image. We propose two deep generative models that generate an image by making use of multiple captions describing it. This is achieved by ensuring {\textquoteleft}Cross-Caption Cycle Consistency{\textquoteright} between the multiple captions and the generated image(s). We report quantitative and qualitative results on the standard Caltech-UCSD Birds (CUB) and Oxford-102 Flowers datasets to validate the efficacy of the proposed approach.",

author = "Joseph, {K. J.} and Arghya Pal and Sailaja Rajanala and Balasubramanian, {Vineeth N.}",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE; IEEE Winter Conference on Applications of Computer Vision 2019, WACV 2019 ; Conference date: 07-01-2019 Through 11-01-2019",

year = "2019",

doi = "10.1109/WACV.2019.00044",

language = "English",

isbn = "9781728119762",

pages = "358--366",

editor = "Michael Brown and Yanxi Liu and Peyman Milanfar",

booktitle = "Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018",

publisher = "IEEE, Institute of Electrical and Electronics Engineers",

address = "United States of America",

url = "https://wacv19.wacv.net/, https://ieeexplore.ieee.org/xpl/conhome/8642793/proceeding",

}

Joseph, KJ, Pal, A , Rajanala, S & Balasubramanian, VN 2019, C4Synth: Cross-caption cycle-consistent text-to-image synthesis. in M Brown, Y Liu & P Milanfar (eds), Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018. IEEE, Institute of Electrical and Electronics Engineers, Piscataway NJ USA, pp. 358-366, IEEE Winter Conference on Applications of Computer Vision 2019, Waikoloa Village, Hawaii, United States of America, 7/01/19. https://doi.org/10.1109/WACV.2019.00044

C4Synth: Cross-caption cycle-consistent text-to-image synthesis. / Joseph, K. J.; Pal, Arghya ; Rajanala, Sailaja et al.
Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018. ed. / Michael Brown; Yanxi Liu; Peyman Milanfar. Piscataway NJ USA: IEEE, Institute of Electrical and Electronics Engineers, 2019. p. 358-366.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - C4Synth

T2 - IEEE Winter Conference on Applications of Computer Vision 2019

AU - Joseph, K. J.

AU - Pal, Arghya

AU - Rajanala, Sailaja

AU - Balasubramanian, Vineeth N.

N1 - Conference code: 19th

PY - 2019

Y1 - 2019

N2 - Generating an image from its description is a challenging task worth solving because of its numerous practical applications ranging from image editing to virtual reality. All existing methods use one single caption to generate a plausible image. A single caption by itself, can be limited and may not be able to capture the variety of concepts and behavior that would be present in the image. We propose two deep generative models that generate an image by making use of multiple captions describing it. This is achieved by ensuring ‘Cross-Caption Cycle Consistency’ between the multiple captions and the generated image(s). We report quantitative and qualitative results on the standard Caltech-UCSD Birds (CUB) and Oxford-102 Flowers datasets to validate the efficacy of the proposed approach.

AB - Generating an image from its description is a challenging task worth solving because of its numerous practical applications ranging from image editing to virtual reality. All existing methods use one single caption to generate a plausible image. A single caption by itself, can be limited and may not be able to capture the variety of concepts and behavior that would be present in the image. We propose two deep generative models that generate an image by making use of multiple captions describing it. This is achieved by ensuring ‘Cross-Caption Cycle Consistency’ between the multiple captions and the generated image(s). We report quantitative and qualitative results on the standard Caltech-UCSD Birds (CUB) and Oxford-102 Flowers datasets to validate the efficacy of the proposed approach.

UR - http://www.scopus.com/inward/record.url?scp=85063577984&partnerID=8YFLogxK

U2 - 10.1109/WACV.2019.00044

DO - 10.1109/WACV.2019.00044

M3 - Conference Paper

AN - SCOPUS:85063577984

SN - 9781728119762

SP - 358

EP - 366

BT - Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision WACV 2018

A2 - Brown, Michael

A2 - Liu, Yanxi

A2 - Milanfar, Peyman

PB - IEEE, Institute of Electrical and Electronics Engineers

CY - Piscataway NJ USA

Y2 - 7 January 2019 through 11 January 2019

ER -

C4Synth: Cross-caption cycle-consistent text-to-image synthesis

Abstract

Conference

Access to Document

Other files and links

Cite this