Explain-then-translate: An analysis on improving program translation with self-generated explanations

Zilu Tang; Mayank Agarwal; Alex Shypula; Bailin Wang; Derry Wijaya; Jie Chen; Yoon Kim

doi:10.18653/v1/2023.findings-emnlp.119

Explain-then-translate: An analysis on improving program translation with self-generated explanations

Zilu Tang, Mayank Agarwal, Alex Shypula, Bailin Wang, Derry Wijaya, Jie Chen, Yoon Kim

Data Science Indonesia

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

7 Citations (Scopus)

Abstract

This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset (Cassano et al., 2022), we find the explanations to be particularly effective in the zero-shot case, improving performance by 12% on average. Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.

Original language	English
Title of host publication	EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023
Editors	Houda Bouamor, Juan Pino, Kalika Bali
Place of Publication	Stroudsburg PA USA
Publisher	Association for Computational Linguistics (ACL)
Pages	1741-1788
Number of pages	48
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.119
Publication status	Published - 2023
Event	Empirical Methods in Natural Language Processing 2023 - , Singapore Duration: 6 Dec 2023 → 10 Dec 2023 https://2023.emnlp.org/ https://aclanthology.org/volumes/2023.findings-emnlp/ (Proceedings) https://aclanthology.org/volumes/2023.emnlp-demo/ (Proceedings)

Conference

Conference	Empirical Methods in Natural Language Processing 2023
Abbreviated title	EMNLP 2023
Country/Territory	Singapore
Period	6/12/23 → 10/12/23
Internet address	https://2023.emnlp.org/ https://aclanthology.org/volumes/2023.findings-emnlp/ (Proceedings) https://aclanthology.org/volumes/2023.emnlp-demo/ (Proceedings)

Access to Document

10.18653/v1/2023.findings-emnlp.119

Cite this

Tang, Z., Agarwal, M., Shypula, A., Wang, B., Wijaya, D., Chen, J., & Kim, Y. (2023). Explain-then-translate: An analysis on improving program translation with self-generated explanations. In H. Bouamor, J. Pino, & K. Bali (Eds.), EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 1741-1788). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.119

Tang, Zilu ; Agarwal, Mayank ; Shypula, Alex et al. / Explain-then-translate : An analysis on improving program translation with self-generated explanations. EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. editor / Houda Bouamor ; Juan Pino ; Kalika Bali. Stroudsburg PA USA : Association for Computational Linguistics (ACL), 2023. pp. 1741-1788

@inproceedings{7b8be3e7db254939b1d0613e14da6fe8,

title = "Explain-then-translate: An analysis on improving program translation with self-generated explanations",

abstract = "This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset (Cassano et al., 2022), we find the explanations to be particularly effective in the zero-shot case, improving performance by 12\% on average. Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.",

author = "Zilu Tang and Mayank Agarwal and Alex Shypula and Bailin Wang and Derry Wijaya and Jie Chen and Yoon Kim",

note = "Funding Information: We thank all anonymous reviewers. We also thank Justin Weisz, Afra Feyza Aky{\"u}rek, Najoung Kim, Aditya Yetetore, Garry Kuwanto, Reuben Tan, and Yuwen Pu for their helpful discussions and suggestions. This project is supported by the MIT-IBM Watson AI lab and DARPA HR001118S0044 (the LwLL program). ZT is supported by Lu Lingzi Fellowship from Boston University. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor. Funding Information: We thank all anonymous reviewers. We also thank Justin Weisz, Afra Feyza Aky{\"u}rek, Najoung Kim, Aditya Yetetore, Garry Kuwanto, Reuben Tan, and Yuwen Pu for their helpful discussions and suggestions. This project is supported by the MIT-IBM Watson AI lab and DARPAHR001118S0044 (the LwLL program). ZT is supported by Lu Lingzi Fellowship from Boston University. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor. Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; Empirical Methods in Natural Language Processing 2023, EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.119",

language = "English",

pages = "1741--1788",

editor = "Houda Bouamor and Juan Pino and Kalika Bali",

booktitle = "EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

url = "https://2023.emnlp.org/, https://aclanthology.org/volumes/2023.findings-emnlp/, https://aclanthology.org/volumes/2023.emnlp-demo/",

}

Tang, Z, Agarwal, M, Shypula, A, Wang, B, Wijaya, D, Chen, J & Kim, Y 2023, Explain-then-translate: An analysis on improving program translation with self-generated explanations. in H Bouamor, J Pino & K Bali (eds), EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), Stroudsburg PA USA, pp. 1741-1788, Empirical Methods in Natural Language Processing 2023, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.119

Explain-then-translate: An analysis on improving program translation with self-generated explanations. / Tang, Zilu; Agarwal, Mayank; Shypula, Alex et al.
EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. ed. / Houda Bouamor; Juan Pino; Kalika Bali. Stroudsburg PA USA: Association for Computational Linguistics (ACL), 2023. p. 1741-1788.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Explain-then-translate

T2 - Empirical Methods in Natural Language Processing 2023

AU - Tang, Zilu

AU - Agarwal, Mayank

AU - Shypula, Alex

AU - Wang, Bailin

AU - Wijaya, Derry

AU - Chen, Jie

AU - Kim, Yoon

N1 - Funding Information: We thank all anonymous reviewers. We also thank Justin Weisz, Afra Feyza Akyürek, Najoung Kim, Aditya Yetetore, Garry Kuwanto, Reuben Tan, and Yuwen Pu for their helpful discussions and suggestions. This project is supported by the MIT-IBM Watson AI lab and DARPA HR001118S0044 (the LwLL program). ZT is supported by Lu Lingzi Fellowship from Boston University. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor. Funding Information: We thank all anonymous reviewers. We also thank Justin Weisz, Afra Feyza Akyürek, Najoung Kim, Aditya Yetetore, Garry Kuwanto, Reuben Tan, and Yuwen Pu for their helpful discussions and suggestions. This project is supported by the MIT-IBM Watson AI lab and DARPAHR001118S0044 (the LwLL program). ZT is supported by Lu Lingzi Fellowship from Boston University. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor. Publisher Copyright: © 2023 Association for Computational Linguistics.

PY - 2023

Y1 - 2023

N2 - This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset (Cassano et al., 2022), we find the explanations to be particularly effective in the zero-shot case, improving performance by 12% on average. Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.

AB - This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset (Cassano et al., 2022), we find the explanations to be particularly effective in the zero-shot case, improving performance by 12% on average. Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.

UR - http://www.scopus.com/inward/record.url?scp=85183298862&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.findings-emnlp.119

DO - 10.18653/v1/2023.findings-emnlp.119

M3 - Conference Paper

AN - SCOPUS:85183298862

SP - 1741

EP - 1788

BT - EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023

A2 - Bouamor, Houda

A2 - Pino, Juan

A2 - Bali, Kalika

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg PA USA

Y2 - 6 December 2023 through 10 December 2023

ER -

Tang Z, Agarwal M, Shypula A, Wang B, Wijaya D, Chen J et al. Explain-then-translate: An analysis on improving program translation with self-generated explanations. In Bouamor H, Pino J, Bali K, editors, EMNLP 2023 - The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. Stroudsburg PA USA: Association for Computational Linguistics (ACL). 2023. p. 1741-1788 doi: 10.18653/v1/2023.findings-emnlp.119

Explain-then-translate: An analysis on improving program translation with self-generated explanations

Abstract

Conference

Access to Document

Other files and links

Cite this