Preconditioning an artificial neural network using Naive Bayes

Nayyar A. Zaidi; François Petitjean; Geoff I. Webb

doi:10.1007/978-3-319-31753-3_28

Preconditioning an artificial neural network using Naive Bayes

Nayyar A. Zaidi, François Petitjean, Geoff I. Webb

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

5 Citations (Scopus)

Abstract

Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

Original language	English
Title of host publication	Advances in Knowledge Discovery and Data Mining
Subtitle of host publication	20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I
Editors	James Bailey, Latifur Khan, Takashi Washio, Gillian Dobbie, Joshua Zhexue Huang
Place of Publication	Switzerland
Publisher	Springer
Pages	341-353
Number of pages	13
Volume	1
ISBN (Electronic)	9783319317533
ISBN (Print)	9783319317526
DOIs	https://doi.org/10.1007/978-3-319-31753-3_28
Publication status	Published - 2016
Event	Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016 - Auckland, New Zealand Duration: 19 Apr 2016 → 22 Apr 2016 Conference number: 20th http://pakdd16.wordpress.fos.auckland.ac.nz/ https://link.springer.com/book/10.1007/978-3-319-31753-3

Publication series

Name	Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)
Publisher	Springer
Volume	9651
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016
Abbreviated title	PAKDD 2016
Country	New Zealand
City	Auckland
Period	19/04/16 → 22/04/16
Internet address	http://pakdd16.wordpress.fos.auckland.ac.nz/ https://link.springer.com/book/10.1007/978-3-319-31753-3

Keywords

Artificial neural networks
Conditional loglikelihood
Logistic regression
Mean-square-error
Preconditioning
WANBIA-C

Access to Document

10.1007/978-3-319-31753-3_28

Link to publication in Scopus

Cite this

Zaidi, N. A., Petitjean, F., & Webb, G. I. (2016). Preconditioning an artificial neural network using Naive Bayes. In J. Bailey, L. Khan, T. Washio, G. Dobbie, & J. Z. Huang (Eds.), Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I (Vol. 1, pp. 341-353). (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science); Vol. 9651). Springer. https://doi.org/10.1007/978-3-319-31753-3_28

Zaidi, Nayyar A. ; Petitjean, François ; Webb, Geoff I. / Preconditioning an artificial neural network using Naive Bayes. Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. editor / James Bailey ; Latifur Khan ; Takashi Washio ; Gillian Dobbie ; Joshua Zhexue Huang. Vol. 1 Switzerland : Springer, 2016. pp. 341-353 (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)).

@inproceedings{3704335de51c4584b5486f1fe998bdc2,

title = "Preconditioning an artificial neural network using Naive Bayes",

abstract = "Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.",

keywords = "Artificial neural networks, Conditional loglikelihood, Logistic regression, Mean-square-error, Preconditioning, WANBIA-C",

author = "Zaidi, {Nayyar A.} and Fran{\c c}ois Petitjean and Webb, {Geoff I.}",

year = "2016",

doi = "10.1007/978-3-319-31753-3_28",

language = "English",

isbn = "9783319317526",

volume = "1",

series = "Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)",

publisher = "Springer",

pages = "341--353",

editor = "James Bailey and Latifur Khan and Takashi Washio and Gillian Dobbie and Huang, {Joshua Zhexue}",

booktitle = "Advances in Knowledge Discovery and Data Mining",

note = "Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016, PAKDD 2016 ; Conference date: 19-04-2016 Through 22-04-2016",

url = "http://pakdd16.wordpress.fos.auckland.ac.nz/, https://link.springer.com/book/10.1007/978-3-319-31753-3",

}

Zaidi, NA, Petitjean, F & Webb, GI 2016, Preconditioning an artificial neural network using Naive Bayes. in J Bailey, L Khan, T Washio, G Dobbie & JZ Huang (eds), Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. vol. 1, Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science), vol. 9651, Springer, Switzerland, pp. 341-353, Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016, Auckland, New Zealand, 19/04/16. https://doi.org/10.1007/978-3-319-31753-3_28

Preconditioning an artificial neural network using Naive Bayes. / Zaidi, Nayyar A.; Petitjean, François; Webb, Geoff I.

Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. ed. / James Bailey; Latifur Khan; Takashi Washio; Gillian Dobbie; Joshua Zhexue Huang. Vol. 1 Switzerland : Springer, 2016. p. 341-353 (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science); Vol. 9651).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Preconditioning an artificial neural network using Naive Bayes

AU - Zaidi, Nayyar A.

AU - Petitjean, François

AU - Webb, Geoff I.

N1 - Conference code: 20th

PY - 2016

Y1 - 2016

N2 - Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

AB - Logistic Regression (LR) is a workhorse of the statistics community and a state-of-the-art machine learning classifier. It learns a linear model from inputs to outputs trained by optimizing the Conditional Log-Likelihood (CLL) of the data. Recently, it has been shown that preconditioning LR using a Naive Bayes (NB) model speeds up LR learning many-fold. One can, however, train a linear model by optimizing the mean-square-error (MSE) instead of CLL. This leads to an Artificial Neural Network (ANN) with no hidden layer. In this work, we study the effect of NB preconditioning on such an ANN classifier. Optimizing MSE instead of CLL may lead to a lower bias classifier and hence result in better performance on big datasets. We show that this NB preconditioning can speed-up convergence significantly. We also show that optimizing a linear model with MSE leads to a lower bias classifier than optimizing with CLL. We also compare the performance to state-of-the-art classifier Random Forest.

KW - Artificial neural networks

KW - Conditional loglikelihood

KW - Logistic regression

KW - Mean-square-error

KW - Preconditioning

KW - WANBIA-C

UR - http://www.scopus.com/inward/record.url?scp=84964077894&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-31753-3_28

DO - 10.1007/978-3-319-31753-3_28

M3 - Conference Paper

AN - SCOPUS:84964077894

SN - 9783319317526

VL - 1

T3 - Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)

SP - 341

EP - 353

BT - Advances in Knowledge Discovery and Data Mining

A2 - Bailey, James

A2 - Khan, Latifur

A2 - Washio, Takashi

A2 - Dobbie, Gillian

A2 - Huang, Joshua Zhexue

PB - Springer

CY - Switzerland

T2 - Pacific-Asia Conference on Knowledge Discovery and Data Mining 2016

Y2 - 19 April 2016 through 22 April 2016

ER -

Zaidi NA, Petitjean F, Webb GI. Preconditioning an artificial neural network using Naive Bayes. In Bailey J, Khan L, Washio T, Dobbie G, Huang JZ, editors, Advances in Knowledge Discovery and Data Mining: 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, Proceedings, Part I. Vol. 1. Switzerland: Springer. 2016. p. 341-353. (Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)). https://doi.org/10.1007/978-3-319-31753-3_28