Unpaired image captioning by language pivoting

Jiuxiang Gu; Shafiq Joty; Jianfei Cai; Gang Wang

doi:10.1007/978-3-030-01246-5_31

Unpaired image captioning by language pivoting

Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Gang Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

3 Citations (Scopus)

Abstract

Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language description. In general, the mapping function is learned from a training set of image-caption pairs. However, for some language, large scale image-caption paired corpus might not be available. We present an approach to this unpaired image captioning problem by language pivoting. Our method can effectively capture the characteristics of an image captioner from the pivot language (Chinese) and align it to the target language (English) using another pivot-target (Chinese-English) sentence parallel corpus. We evaluate our method on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative comparisons against several baseline approaches demonstrate the effectiveness of our method.

Original language	English
Title of host publication	Computer Vision – ECCV 2018
Subtitle of host publication	15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I
Editors	Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, Yair Weiss
Place of Publication	Cham Switzerland
Publisher	Springer
Pages	519-535
Number of pages	17
ISBN (Electronic)	9783030012465
ISBN (Print)	9783030012458
DOIs	https://doi.org/10.1007/978-3-030-01246-5_31
Publication status	Published - 2018
Externally published	Yes
Event	European Conference on Computer Vision 2018 - Munich, Germany Duration: 8 Sep 2018 → 14 Sep 2018 Conference number: 15th https://eccv2018.org/ https://link.springer.com/book/10.1007/978-3-030-01246-5 (Proceedings)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	11205
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2018
Abbreviated title	ECCV 2018
Country	Germany
City	Munich
Period	8/09/18 → 14/09/18
Internet address	https://eccv2018.org/ https://link.springer.com/book/10.1007/978-3-030-01246-5 (Proceedings)

Keywords

Image captioning
Unpaired learning

Access to Document

10.1007/978-3-030-01246-5_31

Link to publication in Scopus

Cite this

Gu, J., Joty, S., Cai, J., & Wang, G. (2018). Unpaired image captioning by language pivoting. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018: 15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I (pp. 519-535). (Lecture Notes in Computer Science; Vol. 11205). Springer. https://doi.org/10.1007/978-3-030-01246-5_31

Gu, Jiuxiang ; Joty, Shafiq ; Cai, Jianfei ; Wang, Gang. / Unpaired image captioning by language pivoting. Computer Vision – ECCV 2018: 15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I. editor / Vittorio Ferrari ; Martial Hebert ; Cristian Sminchisescu ; Yair Weiss. Cham Switzerland : Springer, 2018. pp. 519-535 (Lecture Notes in Computer Science).

@inproceedings{b36e10d66929489cb32629e3c8d8b913,

title = "Unpaired image captioning by language pivoting",

abstract = "Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language description. In general, the mapping function is learned from a training set of image-caption pairs. However, for some language, large scale image-caption paired corpus might not be available. We present an approach to this unpaired image captioning problem by language pivoting. Our method can effectively capture the characteristics of an image captioner from the pivot language (Chinese) and align it to the target language (English) using another pivot-target (Chinese-English) sentence parallel corpus. We evaluate our method on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative comparisons against several baseline approaches demonstrate the effectiveness of our method.",

keywords = "Image captioning, Unpaired learning",

author = "Jiuxiang Gu and Shafiq Joty and Jianfei Cai and Gang Wang",

year = "2018",

doi = "10.1007/978-3-030-01246-5_31",

language = "English",

isbn = "9783030012458",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "519--535",

editor = "Vittorio Ferrari and Martial Hebert and Cristian Sminchisescu and Yair Weiss",

booktitle = "Computer Vision – ECCV 2018",

note = "European Conference on Computer Vision 2018, ECCV 2018 ; Conference date: 08-09-2018 Through 14-09-2018",

url = "https://eccv2018.org/, https://link.springer.com/book/10.1007/978-3-030-01246-5",

}

Gu, J, Joty, S, Cai, J & Wang, G 2018, Unpaired image captioning by language pivoting. in V Ferrari, M Hebert, C Sminchisescu & Y Weiss (eds), Computer Vision – ECCV 2018: 15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I. Lecture Notes in Computer Science, vol. 11205, Springer, Cham Switzerland, pp. 519-535, European Conference on Computer Vision 2018, Munich, Germany, 8/09/18. https://doi.org/10.1007/978-3-030-01246-5_31

Unpaired image captioning by language pivoting. / Gu, Jiuxiang; Joty, Shafiq; Cai, Jianfei; Wang, Gang.

Computer Vision – ECCV 2018: 15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I. ed. / Vittorio Ferrari; Martial Hebert; Cristian Sminchisescu; Yair Weiss. Cham Switzerland : Springer, 2018. p. 519-535 (Lecture Notes in Computer Science; Vol. 11205).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Unpaired image captioning by language pivoting

AU - Gu, Jiuxiang

AU - Joty, Shafiq

AU - Cai, Jianfei

AU - Wang, Gang

N1 - Conference code: 15th

PY - 2018

Y1 - 2018

N2 - Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language description. In general, the mapping function is learned from a training set of image-caption pairs. However, for some language, large scale image-caption paired corpus might not be available. We present an approach to this unpaired image captioning problem by language pivoting. Our method can effectively capture the characteristics of an image captioner from the pivot language (Chinese) and align it to the target language (English) using another pivot-target (Chinese-English) sentence parallel corpus. We evaluate our method on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative comparisons against several baseline approaches demonstrate the effectiveness of our method.

AB - Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language description. In general, the mapping function is learned from a training set of image-caption pairs. However, for some language, large scale image-caption paired corpus might not be available. We present an approach to this unpaired image captioning problem by language pivoting. Our method can effectively capture the characteristics of an image captioner from the pivot language (Chinese) and align it to the target language (English) using another pivot-target (Chinese-English) sentence parallel corpus. We evaluate our method on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative comparisons against several baseline approaches demonstrate the effectiveness of our method.

KW - Image captioning

KW - Unpaired learning

UR - http://www.scopus.com/inward/record.url?scp=85055110136&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01246-5_31

DO - 10.1007/978-3-030-01246-5_31

M3 - Conference Paper

AN - SCOPUS:85055110136

SN - 9783030012458

T3 - Lecture Notes in Computer Science

SP - 519

EP - 535

BT - Computer Vision – ECCV 2018

A2 - Ferrari, Vittorio

A2 - Hebert, Martial

A2 - Sminchisescu, Cristian

A2 - Weiss, Yair

PB - Springer

CY - Cham Switzerland

T2 - European Conference on Computer Vision 2018

Y2 - 8 September 2018 through 14 September 2018

ER -

Gu J, Joty S, Cai J, Wang G. Unpaired image captioning by language pivoting. In Ferrari V, Hebert M, Sminchisescu C, Weiss Y, editors, Computer Vision – ECCV 2018: 15th European Conference Munich, Germany, September 8–14, 2018 Proceedings, Part I. Cham Switzerland: Springer. 2018. p. 519-535. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-030-01246-5_31