On the usefulness of cross-validation for directional forecast evaluation

Christoph Bergmeir, Mauro Costantini, José M. Benítez

Research output: Contribution to journalArticleResearchpeer-review

Abstract

The usefulness of a predictor evaluation framework which combines a blocked cross-validation scheme with directional accuracy measures is investigated. The advantage of using a blocked cross-validation scheme with respect to the standard out-of-sample procedure is that cross-validation yields more precise error estimates of the prediction error since it makes full use of the data. In order to quantify the gain in precision when directional accuracy measures are considered, a Monte Carlo analysis using univariate and multivariate models is provided. The experiments indicate that more precise estimates are obtained with the blocked cross-validation procedure. An application is carried out on forecasting UK interest rate for illustration purposes. The results show that in such a situation with small samples the cross-validation scheme may have considerable advantages over the standard out-of-sample evaluation procedure as it may help to overcome problems induced by the limited information the directional accuracy measures contain due to their binary nature.

Original languageEnglish
Pages (from-to)132-143
Number of pages12
JournalComputational Statistics and Data Analysis
Volume76
DOIs
Publication statusPublished - 2014
Externally publishedYes

Keywords

  • Blocked cross-validation
  • Forecast directional accuracy
  • Linear models
  • Monte Carlo analysis
  • Out-of-sample evaluation

Cite this

@article{4c593f4d3dc44128a346b6f055c1d0fa,
title = "On the usefulness of cross-validation for directional forecast evaluation",
abstract = "The usefulness of a predictor evaluation framework which combines a blocked cross-validation scheme with directional accuracy measures is investigated. The advantage of using a blocked cross-validation scheme with respect to the standard out-of-sample procedure is that cross-validation yields more precise error estimates of the prediction error since it makes full use of the data. In order to quantify the gain in precision when directional accuracy measures are considered, a Monte Carlo analysis using univariate and multivariate models is provided. The experiments indicate that more precise estimates are obtained with the blocked cross-validation procedure. An application is carried out on forecasting UK interest rate for illustration purposes. The results show that in such a situation with small samples the cross-validation scheme may have considerable advantages over the standard out-of-sample evaluation procedure as it may help to overcome problems induced by the limited information the directional accuracy measures contain due to their binary nature.",
keywords = "Blocked cross-validation, Forecast directional accuracy, Linear models, Monte Carlo analysis, Out-of-sample evaluation",
author = "Christoph Bergmeir and Mauro Costantini and Ben{\'i}tez, {Jos{\'e} M.}",
year = "2014",
doi = "10.1016/j.csda.2014.02.001",
language = "English",
volume = "76",
pages = "132--143",
journal = "Computational Statistics and Data Analysis",
issn = "0167-9473",
publisher = "Elsevier",

}

On the usefulness of cross-validation for directional forecast evaluation. / Bergmeir, Christoph; Costantini, Mauro; Benítez, José M.

In: Computational Statistics and Data Analysis, Vol. 76, 2014, p. 132-143.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - On the usefulness of cross-validation for directional forecast evaluation

AU - Bergmeir, Christoph

AU - Costantini, Mauro

AU - Benítez, José M.

PY - 2014

Y1 - 2014

N2 - The usefulness of a predictor evaluation framework which combines a blocked cross-validation scheme with directional accuracy measures is investigated. The advantage of using a blocked cross-validation scheme with respect to the standard out-of-sample procedure is that cross-validation yields more precise error estimates of the prediction error since it makes full use of the data. In order to quantify the gain in precision when directional accuracy measures are considered, a Monte Carlo analysis using univariate and multivariate models is provided. The experiments indicate that more precise estimates are obtained with the blocked cross-validation procedure. An application is carried out on forecasting UK interest rate for illustration purposes. The results show that in such a situation with small samples the cross-validation scheme may have considerable advantages over the standard out-of-sample evaluation procedure as it may help to overcome problems induced by the limited information the directional accuracy measures contain due to their binary nature.

AB - The usefulness of a predictor evaluation framework which combines a blocked cross-validation scheme with directional accuracy measures is investigated. The advantage of using a blocked cross-validation scheme with respect to the standard out-of-sample procedure is that cross-validation yields more precise error estimates of the prediction error since it makes full use of the data. In order to quantify the gain in precision when directional accuracy measures are considered, a Monte Carlo analysis using univariate and multivariate models is provided. The experiments indicate that more precise estimates are obtained with the blocked cross-validation procedure. An application is carried out on forecasting UK interest rate for illustration purposes. The results show that in such a situation with small samples the cross-validation scheme may have considerable advantages over the standard out-of-sample evaluation procedure as it may help to overcome problems induced by the limited information the directional accuracy measures contain due to their binary nature.

KW - Blocked cross-validation

KW - Forecast directional accuracy

KW - Linear models

KW - Monte Carlo analysis

KW - Out-of-sample evaluation

UR - http://www.scopus.com/inward/record.url?scp=84901591119&partnerID=8YFLogxK

U2 - 10.1016/j.csda.2014.02.001

DO - 10.1016/j.csda.2014.02.001

M3 - Article

VL - 76

SP - 132

EP - 143

JO - Computational Statistics and Data Analysis

JF - Computational Statistics and Data Analysis

SN - 0167-9473

ER -