Calhoun, Gray

WP #11002, March 2012

This paper uses dimension asymptotics to study why overfit linear regression models should be compared out-of-sample; we let the number of predictors used by the larger model increase with the number of observations so that their ratio remains uniformly positive. Under this limit theory, the naive Diebold-Mariano-West out-of-sample test can test hypotheses about a key quantity for evaluating forecasting models---a time series analogue to the generalization error---as long as the out-of-sample period is small relative to the total sample size. Moreover, tests that are designed to reject if the larger model is true, such as the usual in-sample Wald and LM tests and also Clark and McCracken's (2001, 2005a), McCracken's (2007) and Clark and West's (2006, 2007) out-of-sample statistics, will choose the larger model too often when the smaller model is more accurate.

JEL Classification: C01, C12, C22, C52, C53

Keywords: Generalization Error, Forecasting, ModelSelection, t-test, Dimension Asymptotics

Full Text