Minimal penalties and the slope heuristics: a survey

Abstract

Birg{é} and Massart proposed in 2001 the slope heuristics as a way to choose optimally from data an unknown multiplicative constant in front of a penalty. It is built upon the notion of minimal penalty, and it has been generalized since to some "minimal-penalty algorithms". This paper reviews the theoretical results obtained for such algorithms, with a self-contained proof in the simplest framework, precise proof ideas for further generalizations, and a few new results. Explicit connections are made with residual-variance estimators-with an original contribution on this topic, showing that for this task the slope heuristics performs almost as well as a residual-based estimator with the best model choice-and some classical algorithms such as L-curve or elbow heuristics, Mallows' C p , and Akaike's FPE. Practical issues are also addressed, including two new practical definitions of minimal-penalty algorithms that are compared on synthetic data to previously-proposed definitions. Finally, several conjectures and open problems are suggested as future research directions.

References in corpus (13)

• Variance estimation in nonparametric regression via the difference sequence method
• Clustering transformed compositional data using K-means, with applications in gene expression and bicycle sharing system data
• Estimator selection: a new method with applications to kernel density estimation
• Model selection by resampling penalization
• Concentration of quadratic forms under a Bernstein moment assumption
• The noise barrier and the large signal bias of the Lasso and other convex estimators
• State-by-state Minimax Adaptive Estimation for Nonparametric Hidden Markov Models
• Numerical performance of Penalized Comparison to Overfitting for multivariate kernel density estimation
• Nonlinear network-based quantitative trait prediction from transcriptomic data
• Optimistic lower bounds for convex regularized least-squares
• Low rank Multivariate regression
• A concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise
• Clustering and Model Selection via Penalized Likelihood for Different-sized Categorical Data Vectors

Cited by in corpus (8)

• Learning with tree tensor networks: complexity estimates and model selection
• Markov Random Geometric Graph (MRGG): A Growth Model for Temporal Dynamic Networks
• Detecting Abrupt Changes in the Presence of Local Fluctuations and Autocorrelated Noise
• A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models
• Tight Risk Bound for High Dimensional Time Series Completion
• On the use of cross-validation for the calibration of the adaptive lasso
• A comprehensive guideline for regularization-path variable selection in high-dimensional Gaussian linear regression
• Ms.FPOP: An Exact and Fast Segmentation Algorithm With a Multiscale Penalty