Graph-induced restricted Boltzmann machines for document modeling

Tu Dinh Nguyen, Truyen Tran, Dinh Phung, Svetha Venkatesh

Research output: Contribution to journalArticleResearchpeer-review

7 Citations (Scopus)


Discovering knowledge from unstructured texts is a central theme in data mining and machine learning. We focus on fast discovery of thematic structures from a corpus. Our approach is based on a versatile probabilistic formulation - the restricted Boltzmann machine (RBM) - where the underlying graphical model is an undirected bipartite graph. Inference is efficient - document representation can be computed with a single matrix projection, making RBMs suitable for massive text corpora available today. Standard RBMs, however, operate on bag-of-words assumption, ignoring the inherent underlying relational structures among words. This results in less coherent word thematic grouping. We introduce graph-based regularization schemes that exploit the linguistic structures, which in turn can be constructed from either corpus statistics or domain knowledge. We demonstrate that the proposed technique improves the group coherence, facilitates visualization, provides means for estimation of intrinsic dimensionality, reduces overfitting, and possibly leads to better classification accuracy.

Original languageEnglish
Pages (from-to)60-75
Number of pages16
JournalInformation Sciences
Publication statusPublished - Jan 2016
Externally publishedYes


  • Document modeling
  • Feature group discovery
  • Restricted Boltzmann machine
  • Topic coherence
  • Word graphs

Cite this