Adaptively scheduled multitask learning: the case of low-resource Neural Machine Translation

Poorya Zaremoodi, Reza Haffari

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

3 Citations (Scopus)

Abstract

Neural Machine Translation (NMT), a data-hungry technology, suffers from the lack of bilingual data in low-resource scenarios. Multitask learning (MTL) can alleviate this issue by injecting inductive biases into NMT, using auxiliary syntactic and semantic tasks. However, an effective training schedule is required to balance the importance of tasks to get the best use of the training signal. The role of training schedule becomes even more crucial in biased-MTL where the goal is to improve one (or a subset) of tasks the most, e.g. translation quality. Current approaches for biased-MTL are based on brittle hand-engineered heuristics that require trial and error, and should be (re-)designed for each learning scenario. To the best of our knowledge, ours is the first work on adaptively and dynamically changing the training schedule in biased-MTL. We propose a rigorous approach for automatically reweighing the training data of the main and auxiliary tasks throughout the training process based on their contributions to the generalisability of the main NMT task. Our experiments on translating from English to Vietnamese/Turkish/Spanish show improvements of up to +1.2 BLEU points, compared to strong baselines. Additionally, our analyses shed light on the dynamic of needs throughout the training of NMT: from syntax to semantic.

Original languageEnglish
Title of host publicationEMNLP-IJCNLP 2019, 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing
Subtitle of host publicationProceedings of the Conference
EditorsJing Jiang, Vincent Ng, Xiaojun Wan
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Pages177–186
Number of pages10
ISBN (Electronic)9781950737901
DOIs
Publication statusPublished - 2019
EventJoint Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing 2019 - Hong Kong, China
Duration: 3 Nov 20197 Nov 2019
Conference number: 9th
https://www.emnlp-ijcnlp2019.org (Website)
https://www.aclweb.org/anthology/volumes/D19-1/ (Proceedings)

Conference

ConferenceJoint Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing 2019
Abbreviated titleEMNLP-IJCNLP 2019
Country/TerritoryChina
CityHong Kong
Period3/11/197/11/19
Internet address

Cite this