Incorporating side information into recurrent neural network language models

Cong Duy Vu Hoang; Gholamreza Haffari; Trevor Cohn

Incorporating side information into recurrent neural network language models

Cong Duy Vu Hoang, Gholamreza Haffari, Trevor Cohn

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

Original language	English
Title of host publication	The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016)
Subtitle of host publication	Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA
Editors	Ani Nenkova, Owen Rambow
Place of Publication	Stroudsburg, PA
Publisher	Association for Computational Linguistics (ACL)
Pages	1250-1255
Number of pages	6
ISBN (Print)	9781941643914
Publication status	Published - 2016
Event	North American Association for Computational Linguistics 2016 - Sheraton San Diego Hotel & Marina, San Diego, United States of America Duration: 12 Jun 2016 → 17 Jun 2016 Conference number: 15th http://naacl.org/naacl-hlt-2016/

Conference

Conference	North American Association for Computational Linguistics 2016
Abbreviated title	NAACL HLT 2016
Country/Territory	United States of America
City	San Diego
Period	12/06/16 → 17/06/16
Internet address	http://naacl.org/naacl-hlt-2016/

Access to Document

http://naacl.org/naacl-hlt-2016/accepted_papers.html

Cite this

Hoang, C. D. V., Haffari, G., & Cohn, T. (2016). Incorporating side information into recurrent neural network language models. In A. Nenkova, & O. Rambow (Eds.), The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016): Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA (pp. 1250-1255). Association for Computational Linguistics (ACL). http://naacl.org/naacl-hlt-2016/accepted_papers.html

Hoang, Cong Duy Vu ; Haffari, Gholamreza ; Cohn, Trevor. / Incorporating side information into recurrent neural network language models. The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016): Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA. editor / Ani Nenkova ; Owen Rambow. Stroudsburg, PA : Association for Computational Linguistics (ACL), 2016. pp. 1250-1255

@inproceedings{ff7c0217b8984da5892baf235e7bbcf6,

title = "Incorporating side information into recurrent neural network language models",

abstract = "Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.",

author = "Hoang, {Cong Duy Vu} and Gholamreza Haffari and Trevor Cohn",

year = "2016",

language = "English",

isbn = "9781941643914",

pages = "1250--1255",

editor = "Nenkova, {Ani } and Rambow, {Owen }",

booktitle = "The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016)",

publisher = "Association for Computational Linguistics (ACL)",

note = "North American Association for Computational Linguistics 2016, NAACL HLT 2016 ; Conference date: 12-06-2016 Through 17-06-2016",

url = "http://naacl.org/naacl-hlt-2016/",

}

Hoang, CDV, Haffari, G & Cohn, T 2016, Incorporating side information into recurrent neural network language models. in A Nenkova & O Rambow (eds), The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016): Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA. Association for Computational Linguistics (ACL), Stroudsburg, PA, pp. 1250-1255, North American Association for Computational Linguistics 2016, San Diego, California, United States of America, 12/06/16. <http://naacl.org/naacl-hlt-2016/accepted_papers.html>

Incorporating side information into recurrent neural network language models. / Hoang, Cong Duy Vu; Haffari, Gholamreza; Cohn, Trevor.
The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016): Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA. ed. / Ani Nenkova; Owen Rambow. Stroudsburg, PA: Association for Computational Linguistics (ACL), 2016. p. 1250-1255.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Incorporating side information into recurrent neural network language models

AU - Hoang, Cong Duy Vu

AU - Haffari, Gholamreza

AU - Cohn, Trevor

N1 - Conference code: 15th

PY - 2016

Y1 - 2016

N2 - Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

AB - Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

UR - http://www.scopus.com/inward/record.url?scp=84994131561&partnerID=8YFLogxK

M3 - Conference Paper

AN - SCOPUS:84994131561

SN - 9781941643914

SP - 1250

EP - 1255

BT - The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016)

A2 - Nenkova, Ani

A2 - Rambow, Owen

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg, PA

T2 - North American Association for Computational Linguistics 2016

Y2 - 12 June 2016 through 17 June 2016

ER -

Hoang CDV, Haffari G, Cohn T. Incorporating side information into recurrent neural network language models. In Nenkova A, Rambow O, editors, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2016): Proceedings of the Conference, June 12-17, 2016, San Diego, California, USA. Stroudsburg, PA: Association for Computational Linguistics (ACL). 2016. p. 1250-1255

Incorporating side information into recurrent neural network language models

Abstract

Conference

Access to Document

Other files and links

Cite this