Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso

Abstract

In exciting new work, Bertsimas et al. (2016) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes that what was thought possible in the statistics community. They presented empirical comparisons of best subset selection with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset selection consistently outperformed both methods in terms of prediction accuracy. Here we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: (a) neither best subset selection nor the lasso uniformly dominate the other, with best subset selection generally performing better in high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; (b) best subset selection and forward stepwise perform quite similarly throughout; (c) the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen, 2007) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and as well as best subset selection in high SNR scenarios.

18 pages main paper, 34 pages supplement

References in corpus (14)

• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nearly unbiased variable selection under minimax concave penalty
• Pathwise coordinate optimization
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
• Improved variable selection with Forward-Lasso adaptive shrinkage

Cited by in corpus (31)

• Optimization Problems for Machine Learning: A Survey
• Sparse Regression: Scalable algorithms and empirical performance
• Feature Learning for Fault Detection in High-Dimensional Condition-Monitoring Signals
• Machine Learning Methods Economists Should Know About
• Rank-one Convexification for Sparse Regression
• Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
• Sparse and Smooth Signal Estimation: Convexification of L0 Formulations
• Generalized Linear Model Regression under Distance-to-set Penalties
• A systematic review of metaheuristics-based and machine learning-driven intrusion detection systems in IoT
• An Extended Newton-type Algorithm for ${\ell }_2$-Regularized Sparse Logistic Regression and Its Efficiency for Classifying Large-scale Datasets
• Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance
• Simultaneous Feature Selection and Outlier Detection with Optimality Guarantees
• Which bridge estimator is optimal for variable selection?
• Predicting B Cell Receptor Substitution Profiles Using Public Repertoire Data
• The Backbone Method for Ultra-High Dimensional Sparse Machine Learning
• Differentially private methods for managing model uncertainty in linear regression models
• MIP-BOOST: Efficient and Effective $L_0$ Feature Selection for Linear Regression
• Sparse Unit-Sum Regression
• Estimation of Switched Markov Polynomial NARX models
• A generalised OMP algorithm for feature selection with application to gene expression data
• A Robust Learning Algorithm for Regression Models Using Distributionally Robust Optimization under the Wasserstein Metric
• Scalable Algorithms for the Sparse Ridge Regression
• Graph Topological Aspects of Granger Causal Network Learning
• Image denoising via K-SVD with primal-dual active set algorithm
• Principal Component Hierarchy for Sparse Quadratic Programs
• A Refined Algorithm for Curve Fitting by Segmented Straight Lines
• Sparse Models for Machine Learning
• Safe Screening Rules for ${\ell }_0$-Regression
• Directing Power Towards Conic Parameter Subspaces
• Tree-Projected Gradient Descent for Estimating Gradient-Sparse Parameters on Graphs
• From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines