Compressed nonparametric language modelling

Ehsan Shareghi, Gholamreza Haffari, Trevor Cohn

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    2 Citations (Scopus)

    Abstract

    Hierarchical Pitman-Yor Process priors are compelling for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney count-based LM smoothing by up to 15%.

    Original languageEnglish
    Title of host publication26th International Joint Conference on Artificial Intelligence, IJCAI 2017
    EditorsCarles Sierra
    Place of PublicationCA USA
    PublisherInternational Joint Conferences on Artificial Intelligence
    Pages2701-2707
    Number of pages7
    ISBN (Electronic)9780999241103
    DOIs
    Publication statusPublished - 2017
    EventInternational Joint Conference on Artificial Intelligence 2017 - Melbourne, Australia
    Duration: 19 Aug 201725 Aug 2017
    Conference number: 26th
    https://ijcai-17.org/

    Conference

    ConferenceInternational Joint Conference on Artificial Intelligence 2017
    Abbreviated titleIJCAI 2017
    CountryAustralia
    CityMelbourne
    Period19/08/1725/08/17
    Internet address

    Cite this

    Shareghi, E., Haffari, G., & Cohn, T. (2017). Compressed nonparametric language modelling. In C. Sierra (Ed.), 26th International Joint Conference on Artificial Intelligence, IJCAI 2017 (pp. 2701-2707). International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2017/376