Minimal penalties and the slope heuristics: a survey

Abstract

Birg{é} and Massart proposed in 2001 the slope heuristics as a way to choose optimally from data an unknown multiplicative constant in front of a penalty. It is built upon the notion of minimal penalty, and it has been generalized since to some "minimal-penalty algorithms". This paper reviews the theoretical results obtained for such algorithms, with a self-contained proof in the simplest framework, precise proof ideas for further generalizations, and a few new results. Explicit connections are made with residual-variance estimators-with an original contribution on this topic, showing that for this task the slope heuristics performs almost as well as a residual-based estimator with the best model choice-and some classical algorithms such as L-curve or elbow heuristics, Mallows' C p , and Akaike's FPE. Practical issues are also addressed, including two new practical definitions of minimal-penalty algorithms that are compared on synthetic data to previously-proposed definitions. Finally, several conjectures and open problems are suggested as future research directions.

References in corpus (30)

• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Risk bounds for statistical learning
• The discriminative functional mixture model for a comparative analysis of bike sharing systems
• A Kernel Multiple Change-point Algorithm via Model Selection
• Variance estimation in nonparametric regression via the difference sequence method
• Data-driven calibration of penalties for least-squares regression
• Pivotal estimation via square-root Lasso in nonparametric regression
• A new perspective on least squares under convex constraint
• Parametric estimation. Finite sample theory
• Clustering transformed compositional data using K-means, with applications in gene expression and bicycle sharing system data
• Estimator selection: a new method with applications to kernel density estimation
• Homogeneity and change-point detection tests for multivariate data using rank statistics
• Model selection by resampling penalization
• Multi-task Regression using Minimal Penalties
• Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii
• Clustering and variable selection for categorical multivariate data
• Concentration of quadratic forms under a Bernstein moment assumption
• The noise barrier and the large signal bias of the Lasso and other convex estimators
• Adapting to Unknown Sparsity by controlling the False Discovery Rate
• State-by-state Minimax Adaptive Estimation for Nonparametric Hidden Markov Models
• Low rank Multivariate regression
• Nonlinear network-based quantitative trait prediction from transcriptomic data
• Numerical performance of Penalized Comparison to Overfitting for multivariate kernel density estimation
• Optimal model selection for density estimation of stationary data under various mixing conditions
• Optimistic lower bounds for convex regularized least-squares
• Optimal kernel selection for density estimation
• High dimensional regression and matrix estimation without tuning parameters
• Clustering and Model Selection via Penalized Likelihood for Different-sized Categorical Data Vectors
• A concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise
• Oracle approach and slope heuristic in context tree estimation

Cited by in corpus (8)

• Learning with tree tensor networks: complexity estimates and model selection
• Markov Random Geometric Graph (MRGG): A Growth Model for Temporal Dynamic Networks
• Detecting Abrupt Changes in the Presence of Local Fluctuations and Autocorrelated Noise
• A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models
• Tight Risk Bound for High Dimensional Time Series Completion
• On the use of cross-validation for the calibration of the adaptive lasso
• A comprehensive guideline for regularization-path variable selection in high-dimensional Gaussian linear regression
• Ms.FPOP: An Exact and Fast Segmentation Algorithm With a Multiscale Penalty