Forecaster performance evaluation with cross-validation and variants

Christoph Bergmeir, José M. Benítez

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

Abstract

In time series prediction, there is currently no consensus for a best practice of how predictors should be compared and evaluated. We investigate this issue through an empirical study. First, we discuss forecast types, error calculation, and error averaging methods in use, and then we focus on model selection procedures. We consider using ordinary cross-validation techniques and the common time series approach of choosing a test set from the end of a series, as well as less common approaches such as non-dependent cross-validation or blocked cross-validation. The study uses different error measures, various machine learning methods, and synthetic time series data. The results indicate that cross-validation can be a useful tool also in time series evaluation. Theoretical problems can be prevented by using it in the blocked form.

Original languageEnglish
Title of host publicationProceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages849-854
Number of pages6
ISBN (Print)9781457716751
DOIs
Publication statusPublished - 2011
Externally publishedYes
EventInternational Conference on Intelligent Systems Designs and Applications 2011 - Cordoba, Spain
Duration: 22 Nov 201124 Nov 2011
Conference number: 11th

Conference

ConferenceInternational Conference on Intelligent Systems Designs and Applications 2011
Abbreviated titleISDA 2011
CountrySpain
CityCordoba
Period22/11/1124/11/11

Keywords

  • blocked cross-validation
  • cross-validation
  • forecaster evaluation
  • time series

Cite this

Bergmeir, C., & Benítez, J. M. (2011). Forecaster performance evaluation with cross-validation and variants. In Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11 (pp. 849-854). [6121763] IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ISDA.2011.6121763
Bergmeir, Christoph ; Benítez, José M. / Forecaster performance evaluation with cross-validation and variants. Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11. IEEE, Institute of Electrical and Electronics Engineers, 2011. pp. 849-854
@inproceedings{01325be1e9474cecb38a8ac1faf51715,
title = "Forecaster performance evaluation with cross-validation and variants",
abstract = "In time series prediction, there is currently no consensus for a best practice of how predictors should be compared and evaluated. We investigate this issue through an empirical study. First, we discuss forecast types, error calculation, and error averaging methods in use, and then we focus on model selection procedures. We consider using ordinary cross-validation techniques and the common time series approach of choosing a test set from the end of a series, as well as less common approaches such as non-dependent cross-validation or blocked cross-validation. The study uses different error measures, various machine learning methods, and synthetic time series data. The results indicate that cross-validation can be a useful tool also in time series evaluation. Theoretical problems can be prevented by using it in the blocked form.",
keywords = "blocked cross-validation, cross-validation, forecaster evaluation, time series",
author = "Christoph Bergmeir and Ben{\'i}tez, {Jos{\'e} M.}",
year = "2011",
doi = "10.1109/ISDA.2011.6121763",
language = "English",
isbn = "9781457716751",
pages = "849--854",
booktitle = "Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States of America",

}

Bergmeir, C & Benítez, JM 2011, Forecaster performance evaluation with cross-validation and variants. in Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11., 6121763, IEEE, Institute of Electrical and Electronics Engineers, pp. 849-854, International Conference on Intelligent Systems Designs and Applications 2011, Cordoba, Spain, 22/11/11. https://doi.org/10.1109/ISDA.2011.6121763

Forecaster performance evaluation with cross-validation and variants. / Bergmeir, Christoph; Benítez, José M.

Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11. IEEE, Institute of Electrical and Electronics Engineers, 2011. p. 849-854 6121763.

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

TY - GEN

T1 - Forecaster performance evaluation with cross-validation and variants

AU - Bergmeir, Christoph

AU - Benítez, José M.

PY - 2011

Y1 - 2011

N2 - In time series prediction, there is currently no consensus for a best practice of how predictors should be compared and evaluated. We investigate this issue through an empirical study. First, we discuss forecast types, error calculation, and error averaging methods in use, and then we focus on model selection procedures. We consider using ordinary cross-validation techniques and the common time series approach of choosing a test set from the end of a series, as well as less common approaches such as non-dependent cross-validation or blocked cross-validation. The study uses different error measures, various machine learning methods, and synthetic time series data. The results indicate that cross-validation can be a useful tool also in time series evaluation. Theoretical problems can be prevented by using it in the blocked form.

AB - In time series prediction, there is currently no consensus for a best practice of how predictors should be compared and evaluated. We investigate this issue through an empirical study. First, we discuss forecast types, error calculation, and error averaging methods in use, and then we focus on model selection procedures. We consider using ordinary cross-validation techniques and the common time series approach of choosing a test set from the end of a series, as well as less common approaches such as non-dependent cross-validation or blocked cross-validation. The study uses different error measures, various machine learning methods, and synthetic time series data. The results indicate that cross-validation can be a useful tool also in time series evaluation. Theoretical problems can be prevented by using it in the blocked form.

KW - blocked cross-validation

KW - cross-validation

KW - forecaster evaluation

KW - time series

UR - http://www.scopus.com/inward/record.url?scp=84857521234&partnerID=8YFLogxK

U2 - 10.1109/ISDA.2011.6121763

DO - 10.1109/ISDA.2011.6121763

M3 - Conference Paper

SN - 9781457716751

SP - 849

EP - 854

BT - Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11

PB - IEEE, Institute of Electrical and Electronics Engineers

ER -

Bergmeir C, Benítez JM. Forecaster performance evaluation with cross-validation and variants. In Proceedings of the 2011 11th International Conference on Intelligent Systems Design and Applications, ISDA'11. IEEE, Institute of Electrical and Electronics Engineers. 2011. p. 849-854. 6121763 https://doi.org/10.1109/ISDA.2011.6121763