Sparse Regression: Scalable algorithms and empirical performance

Abstract

In this paper, we review state-of-the-art methods for feature selection in statistics with an application-oriented eye. Indeed, sparsity is a valuable property and the profusion of research on the topic might have provided little guidance to practitioners. We demonstrate empirically how noise and correlation impact both the accuracy - the number of correct features selected - and the false detection - the number of incorrect features selected - for five methods: the cardinality-constrained formulation, its Boolean relaxation, ${\ell }_1$ regularization and two methods with non-convex penalties. A cogent feature selection method is expected to exhibit a two-fold convergence, namely the accuracy and false detection rate should converge to $1$ and $0$ respectively, as the sample size increases. As a result, proper method should recover all and nothing but true features. Empirically, the integer optimization formulation and its Boolean relaxation are the closest to exhibit this two properties consistently in various regimes of noise and correlation. In addition, apart from the discrete optimization approach which requires a substantial, yet often affordable, computational time, all methods terminate in times comparable with the \verb|glmnet| package for Lasso. We released code for methods that were not publicly implemented. Jointly considered, accuracy, false detection and computational time provide a comprehensive assessment of each feature selection method and shed light on alternatives to the Lasso-regularization which are not as popular in practice yet.

References in corpus (26)

• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nearly unbiased variable selection under minimax concave penalty
• High-dimensional graphs and variable selection with the Lasso
• Pathwise coordinate optimization
• JuMP: A Modeling Language for Mathematical Optimization
• One-step sparse estimates in nonconcave penalized likelihood models
• Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection
• Sure independence screening in generalized linear models with NP-dimensionality
• Coordinate descent algorithms for lasso penalized regression
• Computing in Operations Research using Julia
• Robustness and Regularization of Support Vector Machines
• Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
• Sparse Classification: a scalable discrete optimization perspective
• On the tightness of an SDP relaxation of k-means
• False Discoveries Occur Early on the Lasso Path
• Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms
• Sparse High-Dimensional Regression: Exact Scalable Algorithms and Phase Transitions

Cited by in corpus (12)

• A Scalable Algorithm For Sparse Portfolio Selection
• Sparse Classification: a scalable discrete optimization perspective
• Prediction of hierarchical time series using structured regularization and its application to artificial neural networks
• Robust subset selection
• A unified approach to mixed-integer optimization problems with logical constraints
• The Backbone Method for Ultra-High Dimensional Sparse Machine Learning
• An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions
• Ideal formulations for constrained convex optimization problems with indicator variables
• Polyhedral results and stronger Lagrangean bounds for stable spanning trees
• Iteratively Reweighted ${\ell }_1$-Penalized Robust Regression
• Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms
• Safe Screening Rules for ${\ell }_0$-Regression