Findings of the WMT 2020 shared task on Chat Translation

M. Amin Farajian; António V. Lopes; André F.T. Martins; Sameen Maruf; Gholamreza Haffari

Findings of the WMT 2020 shared task on Chat Translation

M. Amin Farajian, António V. Lopes, André F.T. Martins, Sameen Maruf, Gholamreza Haffari

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

26 Citations (Scopus)

Abstract

We report the results of the first edition of the WMT shared task on Chat Translation. The task consisted of translating bilingual conversational text, in particular customer support chats for the English-German language pair (English agent, German customer). This task varies from the other translation shared tasks, i.e. news and biomedical, mainly due to the fact that the conversations are bilingual, less planned, more informal, and often ungrammatical. Furthermore, such conversations are usually characterized by shorter and simpler sentences and contain more pronouns. We received 14 submissions from 6 participating teams, all of them covering both directions, i.e. En→De for agent utterances and De→En for customer messages. We used automatic metrics (BLEU and TER) for evaluating the translations of both agent and customer messages and human document-level direct assessments to evaluate the agent translations.

Original language	English
Title of host publication	Fifth Conference on Machine Translation, Proceedings of the Conference
Editors	Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andre Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
Place of Publication	Stroudsburg PA USA
Publisher	Association for Computational Linguistics (ACL)
Pages	65-75
Number of pages	11
ISBN (Electronic)	9781948087810
Publication status	Published - 2020
Event	Conference on Machine Translation 2020 - Online, United States of America Duration: 19 Nov 2020 → 20 Nov 2020 Conference number: 5th https://aclanthology.org/volumes/2020.wmt-1/ (Proceedings)

Conference

Conference	Conference on Machine Translation 2020
Abbreviated title	WMT 2020
Country/Territory	United States of America
Period	19/11/20 → 20/11/20
Internet address	https://aclanthology.org/volumes/2020.wmt-1/ (Proceedings)

Access to Document

371690537-oaFinal published version, 384 KB

https://aclanthology.org/2020.wmt-1.3/

Cite this

Farajian, M. A., Lopes, A. V., Martins, A. F. T., Maruf, S., & Haffari, G. (2020). Findings of the WMT 2020 shared task on Chat Translation. In L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-Jussa, C. Federmann, M. Fishel, A. Fraser, Y. Graham, P. Guzman, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, & M. Negri (Eds.), Fifth Conference on Machine Translation, Proceedings of the Conference (pp. 65-75). Association for Computational Linguistics (ACL). https://aclanthology.org/2020.wmt-1.3/

Farajian, M. Amin ; Lopes, António V. ; Martins, André F.T. et al. / Findings of the WMT 2020 shared task on Chat Translation. Fifth Conference on Machine Translation, Proceedings of the Conference. editor / Loic Barrault ; Ondrej Bojar ; Fethi Bougares ; Rajen Chatterjee ; Marta R. Costa-Jussa ; Christian Federmann ; Mark Fishel ; Alexander Fraser ; Yvette Graham ; Paco Guzman ; Barry Haddow ; Matthias Huck ; Antonio Jimeno Yepes ; Philipp Koehn ; Andre Martins ; Makoto Morishita ; Christof Monz ; Masaaki Nagata ; Toshiaki Nakazawa ; Matteo Negri. Stroudsburg PA USA : Association for Computational Linguistics (ACL), 2020. pp. 65-75

@inproceedings{f6c479e93c1246839f8751fce9306dda,

title = "Findings of the WMT 2020 shared task on Chat Translation",

abstract = "We report the results of the first edition of the WMT shared task on Chat Translation. The task consisted of translating bilingual conversational text, in particular customer support chats for the English-German language pair (English agent, German customer). This task varies from the other translation shared tasks, i.e. news and biomedical, mainly due to the fact that the conversations are bilingual, less planned, more informal, and often ungrammatical. Furthermore, such conversations are usually characterized by shorter and simpler sentences and contain more pronouns. We received 14 submissions from 6 participating teams, all of them covering both directions, i.e. En→De for agent utterances and De→En for customer messages. We used automatic metrics (BLEU and TER) for evaluating the translations of both agent and customer messages and human document-level direct assessments to evaluate the agent translations.",

author = "Farajian, {M. Amin} and Lopes, {Ant{\'o}nio V.} and Martins, {Andr{\'e} F.T.} and Sameen Maruf and Gholamreza Haffari",

note = "Funding Information: We would also like to thank Mathieu Giquel and Ulisses Ferreira for all their help and support during the human evaluation phase, as well as Courtney Stankey for helping in coordinating with the evaluators. This work was supported by the P2020 programs MAIA (contract 045909) and Unbabel4EU (contract 042671), by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundac¸{\~a}o para a Ci{\^e}ncia e Tecnologia through contract UID/50008/2019. Funding Information: We would also like to thank Mathieu Giquel and Ulisses Ferreira for all their help and support during the human evaluation phase, as well as Courtney Stankey for helping in coordinating with the evaluators. This work was supported by the P2020 programs MAIA (contract 045909) and Unbabel4EU (contract 042671), by the European Research Council (ERC StG DeepSPIN 758969), and by the Funda??o para a Ci?ncia e Tecnologia through contract UID/50008/2019. Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; Conference on Machine Translation 2020, WMT 2020 ; Conference date: 19-11-2020 Through 20-11-2020",

year = "2020",

language = "English",

pages = "65--75",

editor = "Loic Barrault and Ondrej Bojar and Fethi Bougares and Rajen Chatterjee and Costa-Jussa, {Marta R.} and Christian Federmann and Mark Fishel and Alexander Fraser and Yvette Graham and Paco Guzman and Barry Haddow and Matthias Huck and Yepes, {Antonio Jimeno} and Philipp Koehn and Andre Martins and Makoto Morishita and Christof Monz and Masaaki Nagata and Toshiaki Nakazawa and Matteo Negri",

booktitle = "Fifth Conference on Machine Translation, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

url = "https://aclanthology.org/volumes/2020.wmt-1/",

}

Farajian, MA, Lopes, AV, Martins, AFT, Maruf, S & Haffari, G 2020, Findings of the WMT 2020 shared task on Chat Translation. in L Barrault, O Bojar, F Bougares, R Chatterjee, MR Costa-Jussa, C Federmann, M Fishel, A Fraser, Y Graham, P Guzman, B Haddow, M Huck, AJ Yepes, P Koehn, A Martins, M Morishita, C Monz, M Nagata, T Nakazawa & M Negri (eds), Fifth Conference on Machine Translation, Proceedings of the Conference. Association for Computational Linguistics (ACL), Stroudsburg PA USA, pp. 65-75, Conference on Machine Translation 2020, United States of America, 19/11/20. <https://aclanthology.org/2020.wmt-1.3/>

Findings of the WMT 2020 shared task on Chat Translation. / Farajian, M. Amin; Lopes, António V.; Martins, André F.T. et al.
Fifth Conference on Machine Translation, Proceedings of the Conference. ed. / Loic Barrault; Ondrej Bojar; Fethi Bougares; Rajen Chatterjee; Marta R. Costa-Jussa; Christian Federmann; Mark Fishel; Alexander Fraser; Yvette Graham; Paco Guzman; Barry Haddow; Matthias Huck; Antonio Jimeno Yepes; Philipp Koehn; Andre Martins; Makoto Morishita; Christof Monz; Masaaki Nagata; Toshiaki Nakazawa; Matteo Negri. Stroudsburg PA USA: Association for Computational Linguistics (ACL), 2020. p. 65-75.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Findings of the WMT 2020 shared task on Chat Translation

AU - Farajian, M. Amin

AU - Lopes, António V.

AU - Martins, André F.T.

AU - Maruf, Sameen

AU - Haffari, Gholamreza

N1 - Conference code: 5th

PY - 2020

Y1 - 2020

N2 - We report the results of the first edition of the WMT shared task on Chat Translation. The task consisted of translating bilingual conversational text, in particular customer support chats for the English-German language pair (English agent, German customer). This task varies from the other translation shared tasks, i.e. news and biomedical, mainly due to the fact that the conversations are bilingual, less planned, more informal, and often ungrammatical. Furthermore, such conversations are usually characterized by shorter and simpler sentences and contain more pronouns. We received 14 submissions from 6 participating teams, all of them covering both directions, i.e. En→De for agent utterances and De→En for customer messages. We used automatic metrics (BLEU and TER) for evaluating the translations of both agent and customer messages and human document-level direct assessments to evaluate the agent translations.

AB - We report the results of the first edition of the WMT shared task on Chat Translation. The task consisted of translating bilingual conversational text, in particular customer support chats for the English-German language pair (English agent, German customer). This task varies from the other translation shared tasks, i.e. news and biomedical, mainly due to the fact that the conversations are bilingual, less planned, more informal, and often ungrammatical. Furthermore, such conversations are usually characterized by shorter and simpler sentences and contain more pronouns. We received 14 submissions from 6 participating teams, all of them covering both directions, i.e. En→De for agent utterances and De→En for customer messages. We used automatic metrics (BLEU and TER) for evaluating the translations of both agent and customer messages and human document-level direct assessments to evaluate the agent translations.

UR - http://www.scopus.com/inward/record.url?scp=85111220754&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:85111220754

SP - 65

EP - 75

BT - Fifth Conference on Machine Translation, Proceedings of the Conference

A2 - Barrault, Loic

A2 - Bojar, Ondrej

A2 - Bougares, Fethi

A2 - Chatterjee, Rajen

A2 - Costa-Jussa, Marta R.

A2 - Federmann, Christian

A2 - Fishel, Mark

A2 - Fraser, Alexander

A2 - Graham, Yvette

A2 - Guzman, Paco

A2 - Haddow, Barry

A2 - Huck, Matthias

A2 - Yepes, Antonio Jimeno

A2 - Koehn, Philipp

A2 - Martins, Andre

A2 - Morishita, Makoto

A2 - Monz, Christof

A2 - Nagata, Masaaki

A2 - Nakazawa, Toshiaki

A2 - Negri, Matteo

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg PA USA

T2 - Conference on Machine Translation 2020

Y2 - 19 November 2020 through 20 November 2020

ER -

Farajian MA, Lopes AV, Martins AFT, Maruf S, Haffari G. Findings of the WMT 2020 shared task on Chat Translation. In Barrault L, Bojar O, Bougares F, Chatterjee R, Costa-Jussa MR, Federmann C, Fishel M, Fraser A, Graham Y, Guzman P, Haddow B, Huck M, Yepes AJ, Koehn P, Martins A, Morishita M, Monz C, Nagata M, Nakazawa T, Negri M, editors, Fifth Conference on Machine Translation, Proceedings of the Conference. Stroudsburg PA USA: Association for Computational Linguistics (ACL). 2020. p. 65-75

Findings of the WMT 2020 shared task on Chat Translation

Abstract

Conference

Access to Document

Other files and links

Cite this