A unified approach to model selection and sparse recovery using regularized least squares

Abstract

Model selection and sparse recovery are two important problems for which many regularization methods have been proposed. We study the properties of regularization methods in both problems under the unified framework of regularized least squares with concave penalties. For model selection, we establish conditions under which a regularized least squares estimator enjoys a nonasymptotic property, called the weak oracle property, where the dimensionality can grow exponentially with sample size. For sparse recovery, we present a sufficient condition that ensures the recoverability of the sparsest solution. In particular, we approach both problems by considering a family of penalties that give a smooth homotopy between $L_0$ and $L_1$ penalties. We also propose the sequentially and iteratively reweighted squares (SIRS) algorithm for sparse recovery. Numerical studies support our theoretical results and demonstrate the advantage of our new methods for model selection and sparse recovery.

Published in at http://dx.doi.org/10.1214/09-AOS683 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (19)

• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Simultaneous analysis of Lasso and Dantzig selector
• Nonconcave penalized likelihood with a diverging number of parameters
• One-step sparse estimates in nonconcave penalized likelihood models
• High-dimensional classification using features annealed independence rules
• Variable selection using MM algorithms
• Variable selection in semiparametric regression modeling
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery
• Sharp thresholds for high-dimensional and noisy recovery of sparsity
• Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

Cited by in corpus (33)

• Robust rank correlation based screening
• Tuning parameter selection in high dimensional penalized likelihood
• Manifold Elastic Net: A Unified Framework for Sparse Dimension Reduction
• Adaptive robust variable selection
• Regularization for Cox's proportional hazards model with NP-dimensionality
• Functional additive regression
• The fused Kolmogorov filter: A nonparametric model-free screening method
• Endogeneity in high dimensions
• Variable selection in linear mixed effects models
• Model Selection Principles in Misspecified Models
• Accelerated Schemes for the $L_1/L_2$ Minimization
• Regularization Methods for High-Dimensional Instrumental Variables Regression With an Application to Genetical Genomics
• High-Dimensional Sparse Additive Hazards Regression
• Innovated interaction screening for high-dimensional nonlinear classification
• Nonconcave penalized composite conditional likelihood estimation of sparse Ising models
• High dimensional thresholded regression and shrinkage effect
• A new scope of penalized empirical likelihood with high-dimensional estimating equations
• Parametric or nonparametric? A parametricness index for model selection
• Asymptotic equivalence of regularization methods in thresholded parameter space
• A Novel Regularization Based on the Error Function for Sparse Recovery
• Asymptotic properties for combined $L_1$ and concave regularization
• Minimizing L1 over L2 norms on the gradient
• An Overview of Healthcare Data Analytics With Applications to the COVID-19 Pandemic
• Impacts of high dimensionality in finite samples
• Ultrahigh-dimensional Robust and Efficient Sparse Regression using Non-Concave Penalized Density Power Divergence
• Selection of tuning parameters in bridge regression models via Bayesian information criterion
• High-dimensional Censored Regression via the Penalized Tobit Likelihood
• The springback penalty for robust signal recovery
• Parallel integrative learning for large-scale multi-response regression with incomplete outcomes
• Discussion: "A significance test for the lasso"
• A Generative Approach to Joint Modeling of Quantitative and Qualitative Responses
• Culling the herd of moments with penalized empirical likelihood
• Numerical analysis of non-local calculus on finite weighted graphs, with application to reduced-order modelling of dynamical systems