Consistency of cross validation for comparing regression procedures

Abstract

Theoretical developments on cross validation (CV) have mainly focused on selecting one among a list of finite-dimensional models (e.g., subset or order selection in linear regression) or selecting a smoothing parameter (e.g., bandwidth for kernel smoothing). However, little is known about consistency of cross validation when applied to compare between parametric and nonparametric methods or within nonparametric methods. We show that under some conditions, with an appropriate choice of data splitting ratio, cross validation is consistent in the sense of selecting the better procedure with probability approaching 1. Our results reveal interesting behavior of cross validation. When comparing two models (procedures) converging at the same nonparametric rate, in contrast to the parametric case, it turns out that the proportion of data used for evaluation in CV does not need to be dominating in size. Furthermore, it can even be of a smaller order than the proportion for estimation while not affecting the consistency property.

Published in at http://dx.doi.org/10.1214/009053607000000514 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (26)

• A survey of cross-validation procedures for model selection
• Cross-validation: what does it estimate and how well does it do it?
• Model selection by resampling penalization
• Optimal cross-validation in density estimation with the $L^2$-loss
• Choice of V for V-Fold Cross-Validation in Least-Squares Density Estimation
• Segmentation of the mean of heteroscedastic data via cross-validation
• V-fold cross-validation improved: V-fold penalization
• Parametric or nonparametric? A parametricness index for model selection
• Walking dynamics are symmetric (enough)
• Network cross-validation by edge sampling
• Catching Up Faster by Switching Sooner: A Prequential Solution to the AIC-BIC Dilemma
• Model Selection and Multiple Testing - A Bayesian and Empirical Bayes Overview and some New Results
• Cross-Validation with Confidence
• Consistent Risk Estimation in Moderately High-Dimensional Linear Regression
• A leave-p-out based estimation of the proportion of null hypotheses
• Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation
• Concentration inequalities of the cross-validation estimate for stable predictors
• Cross-Validation, Risk Estimation, and Model Selection
• Model-Twin Randomization (MoTR) for Estimating the Recurring Individual Treatment Effect
• Cross-Validation and Uncertainty Determination for Randomized Neural Networks with Applications to Mobile Sensors
• Sparsity Oriented Importance Learning for High-dimensional Linear Regression
• Statistical Inference on Partially Linear Panel Model under Unobserved Linearity
• Cross Validation for Comparing Multiple Density Estimation Procedures
• Targeted Cross-Validation
• Person as Population: A Longitudinal View of Single-Subject Causal Inference for Analyzing Self-Tracked Health Data
• On the within-family Kullback-Leibler risk in Gaussian Predictive models