Model selection by resampling penalization

Abstract

In this paper, a new family of resampling-based penalization procedures for model selection is defined in a general framework. It generalizes several methods, including Efron's bootstrap penalization and the leave-one-out penalization recently proposed by Arlot (2008), to any exchangeable weighted bootstrap resampling scheme. In the heteroscedastic regression framework, assuming the models to have a particular structure, these resampling penalties are proved to satisfy a non-asymptotic oracle inequality with leading constant close to 1. In particular, they are asympotically optimal. Resampling penalties are used for defining an estimator adapting simultaneously to the smoothness of the regression function and to the heteroscedasticity of the noise. This is remarkable because resampling penalties are general-purpose devices, which have not been built specifically to handle heteroscedastic data. Hence, resampling penalties naturally adapt to heteroscedasticity. A simulation study shows that resampling penalties improve on V-fold cross-validation in terms of final prediction error, in particular when the signal-to-noise ratio is not large.

extended version of hal-00125455, with a technical appendix

References in corpus (11)

• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Local Rademacher complexities
• Consistency of cross validation for comparing regression procedures
• Data-driven calibration of penalties for least-squares regression
• Model selection by resampling penalization
• Complexity regularization via localized random penalties
• V-fold cross-validation improved: V-fold penalization
• Rejoinder: 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Adaptive asymptotically efficient estimation in heteroscedastic nonparametric regression via model selection
• Choosing a penalty for model selection in heteroscedastic regression
• Simultaneous estimation of the mean and the variance in heteroscedastic Gaussian regression

Cited by in corpus (20)

• A survey of cross-validation procedures for model selection
• Data-driven calibration of penalties for least-squares regression
• Model selection by resampling penalization
• Choice of V for V-Fold Cross-Validation in Least-Squares Density Estimation
• Segmentation of the mean of heteroscedastic data via cross-validation
• Minimal penalties and the slope heuristics: a survey
• Adaptive density estimation for stationary processes
• Choosing a penalty for model selection in heteroscedastic regression
• Optimal model selection for density estimation of stationary data under various mixing conditions
• Margin-adaptive model selection in statistical learning
• The Slope Heuristics in Heteroscedastic Regression
• Spatial adaptation in heteroscedastic regression: Propagation approach
• Concentration inequalities of the cross-validation estimate for stable predictors
• Bootstrap tuning in ordered model selection
• Optimal model selection in density estimation
• Concentration inequalities of the cross-validation estimator for Empirical Risk Minimiser
• Model Selection by Loss Rank for Classification and Unsupervised Learning
• Pattern Coding Meets Censoring: (almost) Adaptive Coding on Countable Alphabets
• Estimating Subagging by cross-validation
• Adaptive non-asymptotic confidence balls in density estimation