The Lasso with general Gaussian designs with applications to hypothesis testing

Abstract

The Lasso is a method for high-dimensional regression, which is now commonly used when the number of covariates $p$ is of the same order or larger than the number of observations $n$. Classical asymptotic normality theory does not apply to this model due to two fundamental reasons: $(1)$ The regularized risk is non-smooth; $(2)$ The distance between the estimator $\hat{\mathbf{\theta }}$ and the true parameters vector ${\mathbf{\theta }}^{\ast }$ cannot be neglected. As a consequence, standard perturbative arguments that are the traditional basis for asymptotic normality fail. On the other hand, the Lasso estimator can be precisely characterized in the regime in which both $n$ and $p$ are large and $n/p$ is of order one. This characterization was first obtained in the case of Gaussian designs with i.i.d. covariates: here we generalize it to Gaussian correlated designs with non-singular covariance structure. This is expressed in terms of a simpler ``fixed-design'' model. We establish non-asymptotic bounds on the distance between the distribution of various quantities in the two models, which hold uniformly over signals ${\mathbf{\theta }}^{\ast }$ in a suitable sparsity class and over values of the regularization parameter. As an application, we study the distribution of the debiased Lasso and show that a degrees-of-freedom correction is necessary for computing valid confidence intervals.

final version accepted to Annals of Statistics

References in corpus (11)

• Simultaneous analysis of Lasso and Dantzig selector
• On the "degrees of freedom" of the lasso
• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Universality in polytope phase transitions and message passing algorithms
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Gene Hunting with Knockoffs for Hidden Markov Models
• A framework to characterize performance of LASSO algorithms
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• De-Biasing The Lasso With Degrees-of-Freedom Adjustment
• Fast and Powerful Conditional Randomization Testing via Distillation
• Second order Stein: SURE for SURE and other applications in high-dimensional inference

Cited by in corpus (19)

• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Gaussian Universality of Perceptrons with Random Labels
• A Power Analysis of the Conditional Randomization Test and Knockoffs
• Inference for Heteroskedastic PCA with Missing Data
• Derivatives and residual distribution of regularized M-estimators with application to adaptive tuning
• CAD: Debiasing the Lasso with inaccurate covariate model
• Local convexity of the TAP free energy and AMP convergence for Z2-synchronization
• Out-of-sample error estimate for robust M-estimators with convex penalty
• Randomized tests for high-dimensional regression: A more efficient and powerful solution
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• Tackling small eigen-gaps: Fine-grained eigenvector estimation and inference under heteroscedastic noise
• Efficient Designs of SLOPE Penalty Sequences in Finite Dimension
• Chi-square and normal inference in high-dimensional multi-task regression
• A New Perspective on Debiasing Linear Regressions
• Large dimensional analysis of general margin based classification methods
• Asymptotic normality of robust $M$-estimators with convex penalty
• Characterizing the SLOPE Trade-off: A Variational Perspective and the Donoho-Tanner Limit
• Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective