Preconditioning an artificial neural network using Naive Bayes

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    3 Citations (Scopus)

    Abstract

    Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

    Original languageEnglish
    Title of host publicationAdvances in Knowledge Discovery and Data Mining
    Subtitle of host publication20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I
    EditorsJames Bailey, Latifur Khan, Takashi Washio, Gillian Dobbie, Joshua Zhexue Huang
    Place of PublicationSwitzerland
    PublisherSpringer
    Pages341-353
    Number of pages13
    Volume1
    ISBN (Electronic)9783319317533
    ISBN (Print)9783319317526
    DOIs
    Publication statusPublished - 2016
    EventPacific-Asia Conference on Knowledge Discovery and Data Mining 2016 - Auckland, New Zealand
    Duration: 19 Apr 201622 Apr 2016
    Conference number: 20th
    http://pakdd16.wordpress.fos.auckland.ac.nz/

    Publication series

    NameLecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)
    PublisherSpringer
    Volume9651
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    ConferencePacific-Asia Conference on Knowledge Discovery and Data Mining 2016
    Abbreviated titlePAKDD 2016
    CountryNew Zealand
    CityAuckland
    Period19/04/1622/04/16
    Internet address

    Keywords

    • Artificial neural networks
    • Conditional loglikelihood
    • Logistic regression
    • Mean-square-error
    • Preconditioning
    • WANBIA-C

    Cite this

    Zaidi, N. A., Petitjean, F., & Webb, G. I. (2016). Preconditioning an artificial neural network using Naive Bayes. In J. Bailey, L. Khan, T. Washio, G. Dobbie, & J. Z. Huang (Eds.), Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I (Vol. 1, pp. 341-353). (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science); Vol. 9651). Switzerland: Springer. https://doi.org/10.1007/978-3-319-31753-3_28
    Zaidi, Nayyar A. ; Petitjean, François ; Webb, Geoff I. / Preconditioning an artificial neural network using Naive Bayes. Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. editor / James Bailey ; Latifur Khan ; Takashi Washio ; Gillian Dobbie ; Joshua Zhexue Huang. Vol. 1 Switzerland : Springer, 2016. pp. 341-353 (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)).
    @inproceedings{3704335de51c4584b5486f1fe998bdc2,
    title = "Preconditioning an artificial neural network using Naive Bayes",
    abstract = "Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.",
    keywords = "Artificial neural networks, Conditional loglikelihood, Logistic regression, Mean-square-error, Preconditioning, WANBIA-C",
    author = "Zaidi, {Nayyar A.} and Fran{\cc}ois Petitjean and Webb, {Geoff I.}",
    year = "2016",
    doi = "10.1007/978-3-319-31753-3_28",
    language = "English",
    isbn = "9783319317526",
    volume = "1",
    series = "Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)",
    publisher = "Springer",
    pages = "341--353",
    editor = "James Bailey and Latifur Khan and Takashi Washio and Gillian Dobbie and Huang, {Joshua Zhexue}",
    booktitle = "Advances in Knowledge Discovery and Data Mining",

    }

    Zaidi, NA, Petitjean, F & Webb, GI 2016, Preconditioning an artificial neural network using Naive Bayes. in J Bailey, L Khan, T Washio, G Dobbie & JZ Huang (eds), Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. vol. 1, Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science), vol. 9651, Springer, Switzerland, pp. 341-353, Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016, Auckland, New Zealand, 19/04/16. https://doi.org/10.1007/978-3-319-31753-3_28

    Preconditioning an artificial neural network using Naive Bayes. / Zaidi, Nayyar A.; Petitjean, François; Webb, Geoff I.

    Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. ed. / James Bailey; Latifur Khan; Takashi Washio; Gillian Dobbie; Joshua Zhexue Huang. Vol. 1 Switzerland : Springer, 2016. p. 341-353 (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science); Vol. 9651).

    Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

    TY - GEN

    T1 - Preconditioning an artificial neural network using Naive Bayes

    AU - Zaidi, Nayyar A.

    AU - Petitjean, François

    AU - Webb, Geoff I.

    PY - 2016

    Y1 - 2016

    N2 - Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

    AB - Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

    KW - Artificial neural networks

    KW - Conditional loglikelihood

    KW - Logistic regression

    KW - Mean-square-error

    KW - Preconditioning

    KW - WANBIA-C

    UR - http://www.scopus.com/inward/record.url?scp=84964077894&partnerID=8YFLogxK

    U2 - 10.1007/978-3-319-31753-3_28

    DO - 10.1007/978-3-319-31753-3_28

    M3 - Conference Paper

    AN - SCOPUS:84964077894

    SN - 9783319317526

    VL - 1

    T3 - Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)

    SP - 341

    EP - 353

    BT - Advances in Knowledge Discovery and Data Mining

    A2 - Bailey, James

    A2 - Khan, Latifur

    A2 - Washio, Takashi

    A2 - Dobbie, Gillian

    A2 - Huang, Joshua Zhexue

    PB - Springer

    CY - Switzerland

    ER -

    Zaidi NA, Petitjean F, Webb GI. Preconditioning an artificial neural network using Naive Bayes. In Bailey J, Khan L, Washio T, Dobbie G, Huang JZ, editors, Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. Vol. 1. Switzerland: Springer. 2016. p. 341-353. (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)). https://doi.org/10.1007/978-3-319-31753-3_28