Coordinate descent algorithms for lasso penalized regression

Abstract

Imposition of a lasso penalty shrinks parameter estimates toward zero and performs continuous model selection. Lasso penalized regression is capable of handling linear regression problems where the number of predictors far exceeds the number of cases. This paper tests two exceptionally fast algorithms for estimating regression coefficients with a lasso penalty. The previously known ${\ell }_2$ algorithm is based on cyclic coordinate descent. Our new ${\ell }_1$ algorithm is based on greedy coordinate descent and Edgeworth's algorithm for ordinary ${\ell }_1$ regression. Each algorithm relies on a tuning constant that can be chosen by cross-validation. In some regression problems it is natural to group parameters and penalize parameters group by group rather than separately. If the group penalty is proportional to the Euclidean norm of the parameters of the group, then it is possible to majorize the norm and reduce parameter estimation to ${\ell }_2$ regression with a lasso penalty. Thus, the existing algorithm can be extended to novel settings. Each of the algorithms discussed is tested via either simulated or real data or both. The Appendix proves that a greedy form of the ${\ell }_2$ algorithm converges to the minimum value of the objective function.

Published in at http://dx.doi.org/10.1214/07-AOAS147 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• Pathwise coordinate optimization
• Coordinate descent algorithms for lasso penalized regression

Cited by in corpus (52)

• Challenges of Big Data Analysis
• Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection
• Coordinate descent algorithms for lasso penalized regression
• A Selective Review of Group Selection in High-Dimensional Models
• Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors
• Deep Air Learning: Interpolation, Prediction, and Feature Analysis of Fine-grained Air Quality
• Newtonized Orthogonal Matching Pursuit: Frequency Estimation over the Continuum
• Smoothing proximal gradient method for general structured sparse regression
• Sparse least trimmed squares regression for analyzing high-dimensional large data sets
• Trainable ISTA for Sparse Signal Recovery
• Variable selection and regression analysis for graph-structured covariates with an application to genomics
• Regularization for Cox's proportional hazards model with NP-dimensionality
• Estimation for High-Dimensional Linear Mixed-Effects Models Using ${\ell }_1$-Penalization
• GLMMLasso: An Algorithm for High-Dimensional Generalized Linear Mixed Models Using L1-Penalization
• Doubly Robust Smoothing of Dynamical Processes via Outlier Sparsity Constraints
• Convex Biclustering
• The sparse Laplacian shrinkage estimator for high-dimensional regression
• High-Dimensional Sparse Additive Hazards Regression
• From Sparse Signals to Sparse Residuals for Robust Sensing
• Sparse Regression: Scalable algorithms and empirical performance
• Robust Nonparametric Regression via Sparsity Control with Application to Load Curve Data Cleansing
• Nonconcave penalized composite conditional likelihood estimation of sparse Ising models
• High dimensional thresholded regression and shrinkage effect
• Regression on manifolds: Estimation of the exterior derivative
• A new scope of penalized empirical likelihood with high-dimensional estimating equations
• A majorization-minimization approach to variable selection using spike and slab priors
• Regularized 3D functional regression for brain image data via Haar wavelets
• Inferring sparse Gaussian graphical models with latent structure
• Asymptotic equivalence of regularization methods in thresholded parameter space
• Sparse principal component regression for generalized linear models
• Multicategory vertex discriminant analysis for high-dimensional data
• A fast algorithm for detecting gene-gene interactions in genome-wide association studies
• Robust Parametric Classification and Variable Selection by a Minimum Distance Criterion
• Reconstructing DNA copy number by penalized estimation and imputation
• Sparse Proteomics Analysis - A compressed sensing-based approach for feature selection and classification of high-dimensional proteomics mass spectrometry data
• A Path Algorithm for Constrained Estimation
• MM Algorithms for Minimizing Nonsmoothly Penalized Objective Functions
• Sparse Methods for Automatic Relevance Determination
• Penalized Estimation of Directed Acyclic Graphs From Discrete Data
• Solution Path Clustering with Adaptive Concave Penalty
• Non-Concave Penalization in Linear Mixed-Effects Models and Regularized Selection of Fixed Effects
• Hybrid safe-strong rules for efficient optimization in lasso-type problems
• Faster Subgradient Methods for Functions with Hölderian Growth
• Monte Carlo Simulation for Lasso-Type Problems by Estimator Augmentation
• Scalable Sparse Cox's Regression for Large-Scale Survival Data via Broken Adaptive Ridge
• Parallel integrative learning for large-scale multi-response regression with incomplete outcomes
• Fixed-point and coordinate descent algorithms for regularized kernel methods
• Sparse principal component regression via singular value decomposition approach
• Strong rules for nonconvex penalties and their implications for efficient algorithms in high-dimensional regression
• Iterative Hard Thresholding for Model Selection in Genome-Wide Association Studies
• Massive Parallelization of Massive Sample-size Survival Analysis
• Efficient Sparse Least Absolute Deviation Regression with Differential Privacy