Compressed nonparametric language modelling

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    4 Citations (Scopus)


    Hierarchical Pitman-Yor Process priors are compelling for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney count-based LM smoothing by up to 15%.

    Original languageEnglish
    Title of host publication26th International Joint Conference on Artificial Intelligence, IJCAI 2017
    EditorsCarles Sierra
    Place of PublicationMarina del Rey CA USA
    PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
    Number of pages7
    ISBN (Electronic)9780999241103
    ISBN (Print)9780999241110
    Publication statusPublished - 2017
    EventInternational Joint Conference on Artificial Intelligence 2017 - Melbourne, Australia
    Duration: 19 Aug 201725 Aug 2017
    Conference number: 26th (Proceedings)


    ConferenceInternational Joint Conference on Artificial Intelligence 2017
    Abbreviated titleIJCAI 2017
    Internet address

    Cite this