A latent variable recurrent neural network for discourse relation language models

Yangfeng Ji, Gholamreza Haffari, Jacob Eisenstein

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    77 Citations (Scopus)


    This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.

    Original languageEnglish
    Title of host publicationThe 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016)
    Subtitle of host publicationProceedings of the Conference, June 12-17, 2016, San Diego, California, USA
    EditorsAni Nenkova, Owen Rambow
    Place of PublicationStroudsburg, PA
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages11
    ISBN (Print)9781941643914
    Publication statusPublished - 2016
    EventNorth American Association for Computational Linguistics 2016 - Sheraton San Diego Hotel & Marina, San Diego, United States of America
    Duration: 12 Jun 201617 Jun 2016
    Conference number: 15th


    ConferenceNorth American Association for Computational Linguistics 2016
    Abbreviated titleNAACL HLT 2016
    Country/TerritoryUnited States of America
    CitySan Diego
    Internet address

    Cite this