Asymptotic equivalence of regularization methods in thresholded parameter space
arXiv:1605.03310 · doi:10.1080/01621459.2013.803972
Abstract
High-dimensional data analysis has motivated a spectrum of regularization methods for variable selection and sparse modeling, with two popular classes of convex ones and concave ones. A long debate has been on whether one class dominates the other, an important question both in theory and to practitioners. In this paper, we characterize the asymptotic equivalence of regularization methods, with general penalty functions, in a thresholded parameter space under the generalized linear model setting, where the dimensionality can grow up to exponentially with the sample size. To assess their performance, we establish the oracle inequalities, as in Bickel, Ritov and Tsybakov (2009), of the global minimizer for these methods under various prediction and variable selection losses. These results reveal an interesting phase transition phenomenon. For polynomially growing dimensionality, the -regularization method of Lasso and concave methods are asymptotically equivalent, having the same convergence rates in the oracle inequalities. For exponentially growing dimensionality, concave methods are asymptotically equivalent but have faster convergence rates than the Lasso. We also establish a stronger property of the oracle risk inequalities of the regularization methods, as well as the sampling properties of computable solutions. Our new theoretical results are illustrated and justified by simulation and real data examples.
arXiv comment39 pages, 3 figures
References in corpus (22)
- Discussion of "Least angle regression" by Efron et al
- Rejoinder to "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Least Angle Regression
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Nearly unbiased variable selection under minimax concave penalty
- Simultaneous analysis of Lasso and Dantzig selector
- Pathwise coordinate optimization
- Nonconcave penalized likelihood with a diverging number of parameters
- One-step sparse estimates in nonconcave penalized likelihood models
- Coordinate descent algorithms for lasso penalized regression
- The sparsity and bias of the Lasso selection in high-dimensional linear regression
- High-dimensional classification using features annealed independence rules
- High-dimensional generalized linear models and the lasso
- A unified approach to model selection and sparse recovery using regularized least squares
- Sparsity oracle inequalities for the Lasso
- High-Dimensional Sparse Additive Hazards Regression
Cited by in corpus (4)
- Regularized estimation in sparse high-dimensional time series models
- Innovated interaction screening for high-dimensional nonlinear classification
- Nonparametric independence screening and structure identification for ultra-high dimensional longitudinal data
- Parallel integrative learning for large-scale multi-response regression with incomplete outcomes