On the Selection of Error Measures for Comparisons Among Forecasting Methods

On the Selection of Error Measures for Comparisons Among Forecasting Methods

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
Electronic copy available at: http://ssrn.com/abstract=662661 On the Selection of Error Measures for Comparisons Among Forecasting Methods J. Scott Armstrong University of Pennsylvania, Philadelphia, PA, USA Robert Fildes The Management School, Lancaster University, UK Reprinted with permission from Journal of Forecasting, Vo. 14, 67-71 (1995) ABSTRACT Clements and Hendry (1993) proposed the Generalized Forecast Error Second Moment (GFESM) as an improvement to the Mean Square Error in comparing forecasting performance across data series. They based their conclusion on the fact that rankings based on GFESM remain unaltered if the series are linearly transformed. In this paper, we argue that this evaluation ignores other important criteria. Also, their conclusions were illustrated by a simulation study whose relationship to real data was not obvious. Thirdly, prior empirical studies show that the mean square error is an inappropriate measure to serve as a basis for comparison. This undermines the claims made for the GFESM. KEY WORDS Accuracy Forecast evaluation Loss functions Introduction Presumably, one engages in theoretical manipulations and simulations in order to make generalizations about the real world. The purpose of Clements and Hendry (1993) (henceforth referred to as C&H) was to make generalizations about the best metric for determining which of a set of forecasting methods is expected to be most accurate, They concluded that one should use the Generalized Forecast Error Second Moment (GFESM) rather than the Mean Square Forecast Error (which we refer to as the MSE). The heart of their argument was that whatever error measure is used, it should be invariant to scale-preserving linear transformations. We believe that their conclusion lacks external validity. We discuss three major reasons. First, invariance of rankings to transformations is only one of many criteria that are helpful for examining forecast accuracy. Second, the simulated data were not shown to provide a good representation of real data. Third, empirical studies have shown that mean square errors are inappropriate for the comparison of forecasting methods across different data series (Armstrong and Collopy, 1992; Fildes, 1992). Thus, the MSE is not a good benchmark. Some of these issues are touched on by the discussants of C&H. We highlight what we consider to be the crucial issues, and refer to some relevant empirical studies that were overlooked by C&H and their commentators.
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Electronic copy available at: http://ssrn.com/abstract=662661 Criteria For Adopting Error Measures A forecasting method generates a multivariate error distribution, conditioned by the various forecasting lead times under consideration and the time origin of the forecasts (Murphy and Winkler, 1992). The purpose of an error measure is to provide an informative and clear summary of the distribution. Many commentators, including those who contributed to the discussion of C&H, believe
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 02/28/2010 for the course ECO 211 taught by Professor Gilo during the Spring '10 term at Young Harris.

Page1 / 5

On the Selection of Error Measures for Comparisons Among Forecasting Methods

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online