Correlated variables in regression: clustering and sparse estimation

Abstract

We consider estimation in a high-dimensional linear model with strongly correlated variables. We propose to cluster the variables first and do subsequent sparse estimation such as the Lasso for cluster-representatives or the group Lasso based on the structure from the clusters. Regarding the first step, we present a novel and bottom-up agglomerative clustering algorithm based on canonical correlations, and we show that it finds an optimal solution and is statistically consistent. We also present some theoretical arguments that canonical correlation based clustering leads to a better-posed compatibility constant for the design matrix which ensures identifiability and an oracle inequality for the group Lasso. Furthermore, we discuss circumstances where cluster-representatives and using the Lasso as subsequent estimator leads to improved results for prediction and detection of variables. We complement the theoretical analysis with various empirical results.

40 pages, 6 figures

References in corpus (5)

• One-step sparse estimates in nonconcave penalized likelihood models
• The sparsity and bias of the Lasso selection in high-dimensional linear regression
• Lasso-type recovery of sparse representations for high-dimensional data
• The sparse Laplacian shrinkage estimator for high-dimensional regression
• The Lasso, correlated design, and improved oracle inequalities

Cited by in corpus (29)

• Correlation and variable importance in random forests
• Bayesian linear regression with sparse priors
• On the Prediction Performance of the Lasso
• Group Sparse Recovery via the ${\ell }^0({\ell }^2)$ Penalty: Theory and Algorithm
• Sparse Estimation with Strongly Correlated Variables using Ordered Weighted L1 Regularization
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Extensions of stability selection using subsamples of observations and covariates
• Recursive nearest agglomeration (ReNA): fast clustering for approximation of structured signals
• Concept Tree: High-Level Representation of Variables for More Interpretable Surrogate Decision Trees
• Swapping Variables for High-Dimensional Sparse Regression with Correlated Measurements
• Group-bound: confidence intervals for groups of variables in sparse high-dimensional regression without assumptions on the design
• Graph-based regularization for regression problems with alignment and highly-correlated designs
• Statistical control for spatio-temporal MEG/EEG source imaging with desparsified multi-task Lasso
• Full Optical Fiber Link Characterization with the BSS-Lasso
• Clustering of variables for enhanced interpretability of predictive models
• Iterative Alpha Expansion for estimating gradient-sparse signals from linear measurements
• High-Dimensional Screening Using Multiple Grouping of Variables
• Regularization and Variable Selection with Copula Prior
• Sparse regression and marginal testing using cluster prototypes
• The Cluster Graphical Lasso for improved estimation of Gaussian graphical models
• Efficient Clustering of Correlated Variables and Variable Selection in High-Dimensional Linear Models
• A Two-Stage Variable Selection Approach for Correlated High Dimensional Predictors
• Tree-Projected Gradient Descent for Estimating Gradient-Sparse Parameters on Graphs
• ECKO: Ensemble of Clustered Knockoffs for multivariate inference on fMRI data
• Sparse regression with highly correlated predictors
• Homogeneity and Sub-homogeneity Pursuit: Iterative Complement Clustering PCA
• On the Use of Minimum Penalties in Statistical Learning
• A general framework for estimation and inference from clusters of features
• Fast clustering for scalable statistical analysis on structured images