A fast trust-region Newton method for softmax logistic regression

Nayyar Zaidi, Geoffrey I. Webb

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    1 Citation (Scopus)

    Abstract

    With the emergence of big data, there has been a growing interest in optimization routines that lead to faster convergence of Logistic Regression (LR). Among many optimization methods such as Gradient Descent, Quasi-Newton, Conjugate Gradient, etc., the Trust-region based truncated Newton method (TRON) algorithm has been shown to converge the fastest. The TRON algorithm also forms an important component of the highly efficient and widely used liblinear package. It has been shown that the WANBIA-C trick of scaling with the log of the naive Bayes conditional probabilities can greatly accelerate the convergence of LR trained using (first-order) Gradient Descent and (approximate second-order) Quasi-Newton optimization. In this work we study the applicability of the WANBIA-C trick to TRON. We first devise a TRON algorithm optimizing the softmax objective function and then demonstrate that WANBIA-C style preconditioning can be beneficial for TRON, leading to an extremely fast (batch) LR algorithm. Second, we present a comparative analysis of one-vs-all LR and softmax LR in terms of the 0-1 Loss, Bias, Variance, RMSE, Log-Loss, Training and Classification time, and show that softmax LR leads to significantly better RMSE and Log-Loss. We evaluate our proposed approach on 51 benchmark datasets.

    Original languageEnglish
    Title of host publicationProceedings of the 17th SIAM International Conference on Data Mining
    Subtitle of host publicationHouston, Texas, USA, 27 – 29 April , 2017
    EditorsNitesh Chawla, Wei Wang
    Place of PublicationPhiladelphia, PA
    PublisherSociety for Industrial & Applied Mathematics (SIAM)
    Pages705-713
    Number of pages9
    ISBN (Electronic)9781611974874, 9781611974881
    DOIs
    Publication statusPublished - 2017
    EventSIAM International Conference on Data Mining 2017 - Houston, United States of America
    Duration: 27 Apr 201729 Apr 2017
    Conference number: 17th

    Conference

    ConferenceSIAM International Conference on Data Mining 2017
    Abbreviated titleSDM 2017
    CountryUnited States of America
    CityHouston
    Period27/04/1729/04/17

    Cite this

    Zaidi, N., & Webb, G. I. (2017). A fast trust-region Newton method for softmax logistic regression. In N. Chawla, & W. Wang (Eds.), Proceedings of the 17th SIAM International Conference on Data Mining: Houston, Texas, USA, 27 – 29 April , 2017 (pp. 705-713). Society for Industrial & Applied Mathematics (SIAM). https://doi.org/10.1137/1.9781611974973.79