The composite absolute penalties family for grouped and hierarchical variable selection

Abstract

Extracting useful information from high-dimensional data is an important focus of today's statistical research and practice. Penalized loss function minimization has been shown to be effective for this task both theoretically and empirically. With the virtues of both regularization and sparsity, the $L_1$-penalized squared error minimization method Lasso has been popular in regression models and beyond. In this paper, we combine different norms including $L_1$ to form an intelligent penalty in order to add side information to the fitting of a regression or classification model to obtain reasonable estimates. Specifically, we introduce the Composite Absolute Penalties (CAP) family, which allows given grouping and hierarchical relationships between the predictors to be expressed. CAP penalties are built by defining groups and combining the properties of norm penalties at the across-group and within-group levels. Grouped selection occurs for nonoverlapping groups. Hierarchical variable selection is reached by defining groups with particular overlapping patterns. We propose using the BLASSO and cross-validation to compute CAP estimates in general. For a subfamily of CAP estimates involving only the $L_1$ and $L_{\infty }$ norms, we introduce the iCAP algorithm to trace the entire regularization path for the grouped selection problem. Within this subfamily, unbiased estimates of the degrees of freedom (df) are derived so that the regularization parameter is selected without cross-validation. CAP is shown to improve on the predictive performance of the LASSO in a series of simulated experiments, including cases with $p\gg n$ and possibly mis-specified groupings. When the complexity of a model is properly calculated, iCAP is seen to be parsimonious in the experiments.

Published in at http://dx.doi.org/10.1214/07-AOS584 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (12)

• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• On the "degrees of freedom" of the lasso
• Piecewise linear regularized solution paths

Cited by in corpus (102)

• Sparse Signal Recovery with Temporally Correlated Source Vectors Using Sparse Bayesian Learning
• Structured Variable Selection with Sparsity-Inducing Norms
• Support union recovery in high-dimensional multivariate regression
• A lasso for hierarchical interactions
• A Selective Review of Group Selection in High-Dimensional Models
• A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
• Learning with Structured Sparsity
• Proximal Methods for Hierarchical Sparse Coding
• Smoothing proximal gradient method for general structured sparse regression
• Variable selection for sparse Dirichlet-multinomial regression with an application to microbiome data analysis
• Estimation of Sparse MIMO Channels with Common Support
• Block-Sparse Recovery via Convex Optimization
• Tree-guided group lasso for multi-response regression with structured sparsity, with an application to eQTL mapping
• Network Flow Algorithms for Structured Sparsity
• Structured variable selection and estimation
• Minimax sparse principal subspace estimation in high dimensions
• Smoothing Proximal Gradient Method for General Structured Sparse Learning
• Convex and Network Flow Optimization for Structured Sparsity
• Hybrid Approximate Message Passing
• Nonparametric sparsity and regularization
• Convex Modeling of Interactions with Strong Heredity
• Hierarchical Sparse Modeling: A Choice of Two Group Lasso Formulations
• Variable selection in functional regression models: a review
• Tight conditions for consistency of variable selection in the context of high dimensionality
• Supervised Feature Selection in Graphs with Path Coding Penalties and Network Flows
• Fast Overlapping Group Lasso
• The BerHu penalty and the grouped effect
• Multi-species distribution modeling using penalized mixture of regressions
• Recursive ${\ell }_{1,\infty }$ Group lasso
• On Multiplicative Multitask Feature Learning
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Sparsity with sign-coherent groups of variables via the cooperative-Lasso
• Proximal-Proximal-Gradient Method
• Learning Local Dependence In Ordered Data
• Learning the Structure for Structured Sparsity
• Parallel Direction Method of Multipliers
• Sparse Range-constrained Learning and Its Application for Medical Image Grading
• Learning Efficient Structured Sparse Models
• Exploiting Correlation in Sparse Signal Recovery Problems: Multiple Measurement Vectors, Block Sparsity, and Time-Varying Sparsity
• Learning efficient sparse and low rank models
• Efficient Learning of Sparse Conditional Random Fields for Supervised Sequence Labelling
• Prediction of hierarchical time series using structured regularization and its application to artificial neural networks
• High-dimensional Mixed Graphical Models
• Generalized Conditional Gradient for Sparse Estimation
• Tight conditions for consistent variable selection in high dimensional nonparametric regression
• Reluctant Interaction Modeling
• Structured Estimation in Nonparameteric Cox Model
• Sparse and Switching Infinite Horizon Optimal Control with Mixed-Norm Penalizations
• Main and Interaction Effects Selection for Quadratic Discriminant Analysis via Penalized Linear Regression
• An Adaptive Markov Random Field for Structured Compressive Sensing
• Screening Rules for Overlapping Group Lasso
• Individual Data Protected Integrative Regression Analysis of High-dimensional Heterogeneous Data
• Stochastic Relaxed Inertial Forward-Backward-Forward splitting for Monotone Inclusions in Hilbert spaces
• Efficient algorithm to select tuning parameters in sparse regression modeling with regularization
• Efficient Mixed-Norm Regularization: Algorithms and Safe Screening Methods
• Collaborative Filtering via Group-Structured Dictionary Learning
• Distributed Multitask Learning
• An analysis of penalized interaction models
• A scalable hierarchical lasso for gene-environment interactions
• A Convex-Nonconvex Strategy for Grouped Variable Selection
• Graph-based regularization for regression problems with alignment and highly-correlated designs
• Multimodal Sparse Bayesian Dictionary Learning
• Bayesian group latent factor analysis with structured sparsity
• Collaborative Multi-sensor Classification via Sparsity-based Representation
• Model Selection for High Dimensional Quadratic Regression via Regularization
• Multimodal Multipart Learning for Action Recognition in Depth Videos
• Sparse Partially Linear Additive Models
• Multi-scale Mining of fMRI data with Hierarchical Structured Sparsity
• Hospital Readmission Prediction - Applying Hierarchical Sparsity Norms for Interpretable Models
• The Knowledge Gradient Policy Using A Sparse Additive Belief Model
• P-splines with an l1 penalty for repeated measures
• High-Dimensional Covariance Decomposition into Sparse Markov and Independence Domains
• OMP-type Algorithm with Structured Sparsity Patterns for Multipath Radar Signals
• Structured Input-Output Lasso, with Application to eQTL Mapping, and a Thresholding Algorithm for Fast Estimation
• Convex Banding of the Covariance Matrix
• Strong Sure Screening of Ultra-high Dimensional Data with Interaction Effects
• A Note on High Dimensional Linear Regression with Interactions
• High-Dimensional Covariance Decomposition into Sparse Markov and Independence Models
• Moving Object Detection by Detecting Contiguous Outliers in the Low-Rank Representation
• Fitting ARMA Time Series Models without Identification: A Proximal Approach
• Neural Architecture Search as Sparse Supernet
• Joint Estimation of Precision Matrices in Heterogeneous Populations
• Sparse hierarchical interaction learning with epigraphical projection
• High-dimensional Adaptive Minimax Sparse Estimation with Interactions
• Learning Undirected Graphical Models with Structure Penalty
• Modeling high dimensional multichannel brain signals
• Variable Selection in Regression-based Estimation of Dynamic Treatment Regimes
• A comprehensive guideline for regularization-path variable selection in high-dimensional Gaussian linear regression
• Interquantile Shrinkage in Spatial Quantile Autoregressive Regression models
• Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization
• An Easy-to-Implement Hierarchical Standardization for Variable Selection Under Strong Heredity Constraint
• Structural modeling using overlapped group penalties for discovering predictive biomarkers for subgroup analysis
• Variable selection and structure identification for varying coefficient Cox models
• Fast projections onto mixed-norm balls with applications
• Proximal methods for the latent group lasso penalty
• Parametric Maxflows for Structured Sparse Learning with Convex Relaxations of Submodular Functions
• Regularization for Multiple Kernel Learning via Sum-Product Networks
• Robust Lasso with missing and grossly corrupted observations
• On best subset regression
• Structured Sparse Aggregation
• Pathways-driven Sparse Regression Identifies Pathways and Genes Associated with High-density Lipoprotein Cholesterol in Two Asian Cohorts
• Learning with Optimal Interpolation Norms