A Risk Ratio Comparison of and Penalized Regression
arXiv:1510.06319
Abstract
There has been an explosion of interest in using -regularization in place of -regularization for feature selection. We present theoretical results showing that while -penalized linear regression never outperforms -regularization by more than a constant factor, in some cases using an penalty is infinitely worse than using an penalty. We also show that the "optimal" solutions are often inferior to solutions found using stepwise regression. We also compare algorithms for solving these two problems and show that although solutions can be found efficiently for the problem, the "optimal" solutions are often inferior to solutions found using greedy classic stepwise regression. Furthermore, we show that solutions obtained by solving the convex problem can be improved by selecting the best of the models (for different regularization penalties) by using an criterion. In other words, an approximate solution to the right problem can be better than the exact solution to the wrong problem.
References in corpus (11)
- Rejoinder to "Least angle regression" by Efron et al
- Least Angle Regression
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Empirical Bayes selection of wavelet thresholds
Cited by in corpus (7)
- A Feature Selection Method for Multivariate Performance Measures
- Hybrid quantum-classical optimization for financial index tracking
- Efficient Regularized Regression for Variable Selection with L0 Penalty
- Variable Selection with Second-Generation P-Values
- Lass-0: sparse non-convex regression by local search
- Modelling COVID-19 Pandemic Dynamics Using Transparent, Interpretable, Parsimonious and Simulatable (TIPS) Machine Learning Models: A Case Study from Systems Thinking and System Identification Perspectives
- Towards Ultrahigh Dimensional Feature Selection for Big Data