Incorporating side information into recurrent neural network language models

Cong Duy Vu Hoang, Gholamreza Haffari, Trevor Cohn

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    11 Citations (Scopus)


    Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

    Original languageEnglish
    Title of host publicationThe 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016)
    Subtitle of host publicationProceedings of the Conference, June 12-17, 2016, San Diego, California, USA
    EditorsAni Nenkova, Owen Rambow
    Place of PublicationStroudsburg, PA
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages6
    ISBN (Print)9781941643914
    Publication statusPublished - 2016
    EventNorth American Association for Computational Linguistics 2016: Human Language Technologies - Sheraton San Diego Hotel & Marina, San Diego, United States of America
    Duration: 12 Jun 201617 Jun 2016
    Conference number: 15th


    ConferenceNorth American Association for Computational Linguistics 2016
    Abbreviated titleNAACL HLT 2016
    CountryUnited States of America
    CitySan Diego
    Internet address

    Cite this