Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection

Abstract

A number of variable selection methods have been proposed involving nonconvex penalty functions. These methods, which include the smoothly clipped absolute deviation (SCAD) penalty and the minimax concave penalty (MCP), have been demonstrated to have attractive theoretical properties, but model fitting is not a straightforward task, and the resulting solutions may be unstable. Here, we demonstrate the potential of coordinate descent algorithms for fitting these models, establishing theoretical convergence properties and demonstrating that they are significantly faster than competing approaches. In addition, we demonstrate the utility of convexity diagnostics to determine regions of the parameter space in which the objective function is locally convex, even though the penalty is not. Our simulation study and data examples indicate that nonconvex penalties like MCP and SCAD are worthwhile alternatives to the lasso in many applications. In particular, our numerical results suggest that MCP is the preferred approach among the three methods.

Published in at http://dx.doi.org/10.1214/10-AOAS388 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (14)

• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nearly unbiased variable selection under minimax concave penalty
• Pathwise coordinate optimization
• One-step sparse estimates in nonconcave penalized likelihood models
• Coordinate descent algorithms for lasso penalized regression

Cited by in corpus (62)

• Challenges of Big Data Analysis
• A Selective Review of Group Selection in High-Dimensional Models
• Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
• Tuning parameter selection in high dimensional penalized likelihood
• Regularization for Cox's proportional hazards model with NP-dimensionality
• Feature Selection with Annealing for Computer Vision and Big Data Learning
• Endogeneity in high dimensions
• GLMMLasso: An Algorithm for High-Dimensional Generalized Linear Mixed Models Using L1-Penalization
• Optimal computational and statistical rates of convergence for sparse nonconvex learning problems
• The sparse Laplacian shrinkage estimator for high-dimensional regression
• High-Dimensional Sparse Additive Hazards Regression
• An adaptive Ridge procedure for L0 regularization
• Sparse Regression: Scalable algorithms and empirical performance
• Sparse Estimation by Exponential Weighting
• Efficiency for Regularization Parameter Selection in Penalized Likelihood Estimation of Misspecified Models
• Exploratory Mediation Analysis with Many Potential Mediators
• Vector-Valued Graph Trend Filtering with Non-Convex Penalties
• High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking
• Sparse Classification: a scalable discrete optimization perspective
• Global solutions to folded concave penalized nonconvex learning
• Convex and non-convex regularization methods for spatial point processes intensity estimation
• Estimation of Optimal Individualized Treatment Rules Using a Covariate-Specific Treatment Effect Curve with High-dimensional Covariates
• Sparse PCA with Oracle Property
• A novel algorithm for simultaneous SNP selection in high-dimensional genome-wide association studies
• Fully Bayesian Logistic Regression with Hyper-Lasso Priors for High-dimensional Feature Selection
• Fully Bayesian Classification with Heavy-tailed Priors for Selection in High-dimensional Features with Grouping Structure
• An unbiased approach to compressed sensing
• Cross validation approaches for penalized Cox regression
• High-Dimensional Linear Regression via Implicit Regularization
• Ranked Sparsity: A Cogent Regularization Framework for Selecting and Estimating Feature Interactions and Polynomials
• High-dimensional variable selection via low-dimensional adaptive learning
• Variable selection and estimation for semi-parametric multiple-index models
• Scalable Sparse Cox's Regression for Large-Scale Survival Data via Broken Adaptive Ridge
• A Scalable Empirical Bayes Approach to Variable Selection in Generalized Linear Models
• Sparse Signal Reconstruction for Nonlinear Models via Piecewise Rational Optimization
• A study on tuning parameter selection for the high-dimensional lasso
• On complexity and convergence of high-order coordinate descent algorithms for smooth nonconvex box-constrained minimization
• Strong rules for nonconvex penalties and their implications for efficient algorithms in high-dimensional regression
• Inference for a Large Directed Acyclic Graph with Unspecified Interventions
• Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Iterative Hard Thresholding for Model Selection in Genome-Wide Association Studies
• Feature-specific inference for penalized regression using local false discovery rates
• Massive Parallelization of Massive Sample-size Survival Analysis
• Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration
• A Sharper Computational Tool for ${\text{L}}_2\text{E}$ Regression
• A Laplace Mixture Representation of the Horseshoe and Some Implications
• A Discussion on Practical Considerations with Sparse Regression Methodologies
• MM for Penalized Estimation
• Analyzing Risk Factors for Post-Acute Recovery in Older Adults with Alzheimer's Disease and Related Dementia: A New Semi-Parametric Model for Large-Scale Medicare Claims
• Marginal false discovery rates for penalized regression models
• Regularized Nonlinear Regression with Dependent Errors and its Application to a Biomechanical Model
• Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
• Fast, effective, and coherent time series modeling using the sparsity-ranked lasso
• Accelerated Gradient Methods for Sparse Statistical Learning with Nonconvex Penalties
• Variable Selection Using Relative Importance Rankings
• A novel nonconvex, smooth-at-origin penalty for statistical learning
• Efficient computation of high-dimensional penalized piecewise constant hazard random effects models
• Cox reduction and confidence sets of models: a theoretical elucidation
• Adaptive Bi-Level Variable Selection of Conditional Main Effects for Generalized Linear Models
• Semiparametric Wavelet-based JPEG IV Estimator for endogenously truncated data
• A Multi-Omics Framework for Survival Mediation Analysis of High-Dimensional Proteogenomic Data