Data-driven calibration of penalties for least-squares regression

Abstract

Penalization procedures often suffer from their dependence on multiplying factors, whose optimal values are either unknown or hard to estimate from the data. We propose a completely data-driven calibration algorithm for this parameter in the least-squares regression framework, without assuming a particular shape for the penalty. Our algorithm relies on the concept of minimal penalty, recently introduced by Birge and Massart (2007) in the context of penalized least squares for Gaussian homoscedastic regression. On the positive side, the minimal penalty can be evaluated from the data themselves, leading to a data-driven estimation of an optimal penalty which can be used in practice; on the negative side, their approach heavily relies on the homoscedastic Gaussian nature of their stochastic framework. The purpose of this paper is twofold: stating a more general heuristics for designing a data-driven penalty (the slope heuristics) and proving that it works for penalized least-squares regression with a random design, even for heteroscedastic non-Gaussian data. For technical reasons, some exact mathematical results will be proved only for regressogram bin-width selection. This is at least a first step towards further results, since the approach and the method that we use are indeed general.

References in corpus (3)

• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Model selection by resampling penalization
• V-fold cross-validation improved: V-fold penalization

Cited by in corpus (39)

• Adaptive estimation for Hawkes processes; application to genome analysis
• A Kernel Multiple Change-point Algorithm via Model Selection
• Stochastic epidemics in a homogeneous community
• Stochastic epidemics in a heterogeneous community (Part III of the book Stochastic Epidemic Models and Inference)
• Statistical inference for epidemic processes in a homogeneous community (Part IV of the book Stochastic Epidemic Models and Inference)
• Estimator selection: a new method with applications to kernel density estimation
• Optimal cross-validation in density estimation with the $L^2$-loss
• Model selection by resampling penalization
• Multi-task Regression using Minimal Penalties
• Minimal penalties and the slope heuristics: a survey
• Clustering and variable selection for categorical multivariate data
• Sharp oracle inequalities and slope heuristic for specification probabilities estimation in discrete random fields
• Adaptive density estimation for stationary processes
• Choosing a penalty for model selection in heteroscedastic regression
• Margin-adaptive model selection in statistical learning
• Change Points via Probabilistically Pruned Objectives
• Nonlinear network-based quantitative trait prediction from transcriptomic data
• Optimal model selection for density estimation of stationary data under various mixing conditions
• Spatial adaptation in heteroscedastic regression: Propagation approach
• The Slope Heuristics in Heteroscedastic Regression
• Risk Bounds for CART Classifiers under a Margin Condition
• Local bandwidth selection for kernel density estimation in bifurcating Markov chain model
• AIC, Cp and estimators of loss for elliptically symmetric distributions
• Simultaneous Dimension Reduction and Clustering via the NMF-EM Algorithm
• A concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise
• Bandwidth selection in kernel empirical risk minimization via the gradient
• Linear time dynamic programming for the exact path of optimal models selected from a finite set
• Optimal model selection in density estimation
• Oracle approach and slope heuristic in context tree estimation
• Consistent model selection criteria and goodness-of-fit test for affine causal processes
• Minimum discrepancy principle strategy for choosing $k$ in $k$-NN regression
• An Oracle Approach for Interaction Neighborhood Estimation in Random Fields
• Adaptive non-asymptotic confidence balls in density estimation
• Non-parametric adaptive estimation of the drift for a jump diffusion process
• Nonparametric estimation of jump rates for a specific class of Piecewise Deterministic Markov Processes
• High-dimensional regression with unknown variance
• A new segmentation method for the homogenisation of GNSS-derived IWV time-series
• Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?
• A breakpoint detection in the mean model with heterogeneous variance on fixed time-intervals