One-step sparse estimates in nonconcave penalized likelihood models

Abstract

Fan and Li propose a family of variable selection methods via penalized likelihood using concave penalty functions. The nonconcave penalized likelihood estimators enjoy the oracle properties, but maximizing the penalized likelihood function is computationally challenging, because the objective function is nondifferentiable and nonconcave. In this article, we propose a new unified algorithm based on the local linear approximation (LLA) for maximizing the penalized likelihood for a broad class of concave penalty functions. Convergence and other theoretical properties of the LLA algorithm are established. A distinguished feature of the LLA algorithm is that at each LLA step, the LLA estimator can naturally adopt a sparse representation. Thus, we suggest using the one-step LLA estimator from the LLA algorithm as the final estimates. Statistically, we show that if the regularization parameter is appropriately chosen, the one-step LLA estimates enjoy the oracle properties with good initial estimators. Computationally, the one-step LLA estimation methods dramatically reduce the computational cost in maximizing the nonconcave penalized likelihood. We conduct some Monte Carlo simulation to assess the finite sample performance of the one-step sparse estimation methods. The results are very encouraging.

This paper discussed in: [arXiv:0808.1013], [arXiv:0808.1016], [arXiv:0808.1025]. Rejoinder in [arXiv:0808.1030]. Published in at http://dx.doi.org/10.1214/009053607000000802 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (17)

• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nonconcave penalized likelihood with a diverging number of parameters
• Piecewise linear regularized solution paths
• Variable selection using MM algorithms
• Variable selection in semiparametric regression modeling
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery
• Local partial-likelihood estimation for lifetime data
• Hazard models with varying coefficients for multivariate failure time data

Cited by in corpus (67)

• Nearly unbiased variable selection under minimax concave penalty
• Challenges of Big Data Analysis
• Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection
• Sparse Regularization via Convex Analysis
• Sparsistency and rates of convergence in large covariance matrix estimation
• Model Selection in Linear Mixed Models
• Strong oracle optimality of folded concave penalized estimation
• Robust rank correlation based screening
• Network exploration via the adaptive LASSO and SCAD penalties
• New efficient estimation and variable selection methods for semiparametric varying-coefficient partially linear models
• A Selective Review of Group Selection in High-Dimensional Models
• Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
• A unified approach to model selection and sparse recovery using regularized least squares
• Least angle and ${\ell }_1$ penalized regression: A review
• Sparse Signal Estimation by Maximally Sparse Convex Optimization
• Penalized Composite Quasi-Likelihood for Ultrahigh-Dimensional Variable Selection
• Group-Lasso on Splines for Spectrum Cartography
• Estimating time-varying networks
• Correlated variables in regression: clustering and sparse estimation
• Coauthorship and Citation Networks for Statisticians
• Functional additive regression
• Total Variation Denoising via the Moreau Envelope
• Endogeneity in high dimensions
• Variable selection in linear mixed effects models
• High-Dimensional Sparse Additive Hazards Regression
• Gemini: Graph estimation with matrix variate normal instances
• From Sparse Signals to Sparse Residuals for Robust Sensing
• A Smooth and Locally Sparse Estimator for Functional Linear Regression via Functional SCAD Penalty
• Variable selection in measurement error models
• An adaptive Ridge procedure for L0 regularization
• Sparse Regression: Scalable algorithms and empirical performance
• Least Squares Approximation for a Distributed System
• Penalized maximum likelihood estimation and variable selection in geostatistics
• Change-point detection in high-dimensional covariance structure
• Enhanced Sparsity by Non-Separable Regularization
• Penalized variable selection procedure for Cox models with semiparametric relative risk
• Are Discoveries Spurious? Distributions of Maximum Spurious Correlations and Their Applications
• A majorization-minimization approach to variable selection using spike and slab priors
• Oracle Estimation of a Change Point in High Dimensional Quantile Regression
• Asymptotic equivalence of regularization methods in thresholded parameter space
• Functional response additive model estimation with online virtual stock markets
• Convex and non-convex regularization methods for spatial point processes intensity estimation
• A fast algorithm for detecting gene-gene interactions in genome-wide association studies
• Distributed recovery of jointly sparse signals under communication constraints
• On the estimation of variance parameters in non-standard generalised linear mixed models: Application to penalised smoothing
• ATASI-Net: An Efficient Sparse Reconstruction Network for Tomographic SAR Imaging with Adaptive Threshold
• MM Algorithms for Minimizing Nonsmoothly Penalized Objective Functions
• A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques
• Expandable Factor Analysis
• A review of regularised estimation methods and cross-validation in spatiotemporal statistics
• Ultrahigh-dimensional Robust and Efficient Sparse Regression using Non-Concave Penalized Density Power Divergence
• Monte Carlo Simulation for Lasso-Type Problems by Estimator Augmentation
• An unbiased approach to compressed sensing
• Selection of tuning parameters in bridge regression models via Bayesian information criterion
• Smoothing ${\ell }_1$-penalized estimators for high-dimensional time-course data
• A Model-free Variable Screening Method Based on Leverage Score
• High-Dimensional Composite Quantile Regression: Optimal Statistical Guarantees and Fast Algorithms
• High-dimensional Censored Regression via the Penalized Tobit Likelihood
• Strong rules for nonconvex penalties and their implications for efficient algorithms in high-dimensional regression
• M-estimation in high-dimensional linear model
• Penalized Sparse Covariance Regression with High Dimensional Covariates
• An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models
• Partial Correlation Graphical LASSO
• On the Distribution of the Adaptive LASSO Estimator
• Nonconvex penalized multitask regression using data depth-based penalties
• Compound Poisson Processes, Latent Shrinkage Priors and Bayesian Nonconvex Penalization
• A Weibull Mixture Cure Frailty Model for High-dimensional Covariates