Selective attention for context-aware Neural Machine Translation

Sameen Maruf, André F. T. Martins, Gholamreza Haffari

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

31 Citations (Scopus)


Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentences as context and may not scale to entire documents. To this end, we propose a novel and scalable top-down approach to hierarchical attention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose single-level attention approaches based on sentence or word-level information in the context. The document-level context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depending on whether we use monolingual or bilingual context. Our experiments and evaluation on English-German datasets in different document MT settings show that our selective attention approach not only significantly outperforms context-agnostic baselines but also surpasses context-aware baselines in most cases.

Original languageEnglish
Title of host publicationNAACL 2019, The 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Subtitle of host publicationProceedings of the Conference Vol. 1 (Long and Short Papers), June 2 - June 7, 2019
EditorsChristy Doran, Thamar Solorio
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Number of pages11
ISBN (Electronic)9781950737130
Publication statusPublished - Jun 2019
EventNorth American Association for Computational Linguistics 2019: Human Language Technologies - Minneapolis, United States of America
Duration: 2 Jun 20197 Jun 2019


ConferenceNorth American Association for Computational Linguistics 2019
Abbreviated titleNAACL HLT 2019
Country/TerritoryUnited States of America
Internet address

Cite this