Adaptive online extreme learning machine by regulating forgetting factor by concept drift map

Hualong Yu, Geoffrey I. Webb

Research output: Contribution to journalArticleResearchpeer-review

Abstract

In online-learning, the data is incrementally received and the distributions from which it is drawn may keep changing over time. This phenomenon is widely known as concept drift. Such changes may affect the generalization of a learned model to future data. This problem may be exacerbated by the form of the drift itself changing over time. Quantitative measures to describe and analyze the concept drift have been proposed in previous work. A description composed from these measures is called a concept drift map. We believe that these maps could be useful for guiding how much knowledge in the old model should be forgotten. Therefore, this paper presents an adaptive online learning model that uses a concept drift map to regulate the forgetting factor of an extreme learning machine. Specifically, when a batch of new instances are labeled, the distribution of each class on each attribute is firstly estimated, and then it is compared with the distribution estimated in the previous batch to calculate the magnitude of concept drift, which is further used to regulate the forgetting factor and to update the learning model. Therefore, the novelty of this paper lies in that a quantitative distance metric between two distributions constructed on continuous attribute space is presented to construct concept drift map which can be further associated with the forgetting factor to make the learning model adapt the concept drift. Experimental results on several benchmark stream data sets show the proposed model is generally superior to several previous algorithms when classifying a variety of data streams subject to drift, indicating its effectiveness and feasibility.

Original languageEnglish
Pages (from-to)141-153
Number of pages13
JournalNeurocomputing
DOIs
Publication statusPublished - 28 May 2019

Keywords

  • Concept drift map
  • Extreme learning machine
  • Forgetting factor
  • Online extreme learning machine
  • Online learning

Cite this

@article{8ebadfbc0bcf4eeebff6be32b1c963e9,
title = "Adaptive online extreme learning machine by regulating forgetting factor by concept drift map",
abstract = "In online-learning, the data is incrementally received and the distributions from which it is drawn may keep changing over time. This phenomenon is widely known as concept drift. Such changes may affect the generalization of a learned model to future data. This problem may be exacerbated by the form of the drift itself changing over time. Quantitative measures to describe and analyze the concept drift have been proposed in previous work. A description composed from these measures is called a concept drift map. We believe that these maps could be useful for guiding how much knowledge in the old model should be forgotten. Therefore, this paper presents an adaptive online learning model that uses a concept drift map to regulate the forgetting factor of an extreme learning machine. Specifically, when a batch of new instances are labeled, the distribution of each class on each attribute is firstly estimated, and then it is compared with the distribution estimated in the previous batch to calculate the magnitude of concept drift, which is further used to regulate the forgetting factor and to update the learning model. Therefore, the novelty of this paper lies in that a quantitative distance metric between two distributions constructed on continuous attribute space is presented to construct concept drift map which can be further associated with the forgetting factor to make the learning model adapt the concept drift. Experimental results on several benchmark stream data sets show the proposed model is generally superior to several previous algorithms when classifying a variety of data streams subject to drift, indicating its effectiveness and feasibility.",
keywords = "Concept drift map, Extreme learning machine, Forgetting factor, Online extreme learning machine, Online learning",
author = "Hualong Yu and Webb, {Geoffrey I.}",
year = "2019",
month = "5",
day = "28",
doi = "10.1016/j.neucom.2018.11.098",
language = "English",
pages = "141--153",
journal = "Neurocomputing",
issn = "0925-2312",
publisher = "Elsevier",

}

Adaptive online extreme learning machine by regulating forgetting factor by concept drift map. / Yu, Hualong; Webb, Geoffrey I.

In: Neurocomputing, 28.05.2019, p. 141-153.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Adaptive online extreme learning machine by regulating forgetting factor by concept drift map

AU - Yu, Hualong

AU - Webb, Geoffrey I.

PY - 2019/5/28

Y1 - 2019/5/28

N2 - In online-learning, the data is incrementally received and the distributions from which it is drawn may keep changing over time. This phenomenon is widely known as concept drift. Such changes may affect the generalization of a learned model to future data. This problem may be exacerbated by the form of the drift itself changing over time. Quantitative measures to describe and analyze the concept drift have been proposed in previous work. A description composed from these measures is called a concept drift map. We believe that these maps could be useful for guiding how much knowledge in the old model should be forgotten. Therefore, this paper presents an adaptive online learning model that uses a concept drift map to regulate the forgetting factor of an extreme learning machine. Specifically, when a batch of new instances are labeled, the distribution of each class on each attribute is firstly estimated, and then it is compared with the distribution estimated in the previous batch to calculate the magnitude of concept drift, which is further used to regulate the forgetting factor and to update the learning model. Therefore, the novelty of this paper lies in that a quantitative distance metric between two distributions constructed on continuous attribute space is presented to construct concept drift map which can be further associated with the forgetting factor to make the learning model adapt the concept drift. Experimental results on several benchmark stream data sets show the proposed model is generally superior to several previous algorithms when classifying a variety of data streams subject to drift, indicating its effectiveness and feasibility.

AB - In online-learning, the data is incrementally received and the distributions from which it is drawn may keep changing over time. This phenomenon is widely known as concept drift. Such changes may affect the generalization of a learned model to future data. This problem may be exacerbated by the form of the drift itself changing over time. Quantitative measures to describe and analyze the concept drift have been proposed in previous work. A description composed from these measures is called a concept drift map. We believe that these maps could be useful for guiding how much knowledge in the old model should be forgotten. Therefore, this paper presents an adaptive online learning model that uses a concept drift map to regulate the forgetting factor of an extreme learning machine. Specifically, when a batch of new instances are labeled, the distribution of each class on each attribute is firstly estimated, and then it is compared with the distribution estimated in the previous batch to calculate the magnitude of concept drift, which is further used to regulate the forgetting factor and to update the learning model. Therefore, the novelty of this paper lies in that a quantitative distance metric between two distributions constructed on continuous attribute space is presented to construct concept drift map which can be further associated with the forgetting factor to make the learning model adapt the concept drift. Experimental results on several benchmark stream data sets show the proposed model is generally superior to several previous algorithms when classifying a variety of data streams subject to drift, indicating its effectiveness and feasibility.

KW - Concept drift map

KW - Extreme learning machine

KW - Forgetting factor

KW - Online extreme learning machine

KW - Online learning

UR - http://www.scopus.com/inward/record.url?scp=85061175563&partnerID=8YFLogxK

U2 - 10.1016/j.neucom.2018.11.098

DO - 10.1016/j.neucom.2018.11.098

M3 - Article

SP - 141

EP - 153

JO - Neurocomputing

JF - Neurocomputing

SN - 0925-2312

ER -