Pivotal estimation via square-root Lasso in nonparametric regression

Abstract

We propose a self-tuning $\sqrt{\mathrm{Lasso}}$ method that simultaneously resolves three important practical problems in high-dimensional regression analysis, namely it handles the unknown scale, heteroscedasticity and (drastic) non-Gaussianity of the noise. In addition, our analysis allows for badly behaved designs, for example, perfectly collinear regressors, and generates sharp bounds even in extreme cases, such as the infinite variance case and the noiseless case, in contrast to Lasso. We establish various nonasymptotic bounds for $\sqrt{\mathrm{Lasso}}$ including prediction norm rate and sparsity. Our analysis is based on new impact factors that are tailored for bounding prediction norm. In order to cover heteroscedastic non-Gaussian noise, we rely on moderate deviation theory for self-normalized sums to achieve Gaussian-like results under weak conditions. Moreover, we derive bounds on the performance of ordinary least square (ols) applied to the model selected by $\sqrt{\mathrm{Lasso}}$ accounting for possible misspecification of the selected model. Under mild conditions, the rate of convergence of ols post $\sqrt{\mathrm{Lasso}}$ is as good as $\sqrt{\mathrm{Lasso}}$'s rate. As an application, we consider the use of $\sqrt{\mathrm{Lasso}}$ and ols post $\sqrt{\mathrm{Lasso}}$ as estimators of nuisance parameters in a generic semiparametric problem (nonlinear moment condition or $Z$-problem), resulting in a construction of $\sqrt{n}$-consistent and asymptotically normal estimators of the main parameters.

Published in at http://dx.doi.org/10.1214/14-AOS1204 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (17)

• Templates for Convex Cone Problems with Applications to Sparse Signal Recovery
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
• The sparsity and bias of the Lasso selection in high-dimensional linear regression
• Lasso-type recovery of sparse representations for high-dimensional data
• High-dimensional generalized linear models and the lasso
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors
• Near-ideal model selection by ${\ell }_1$ minimization
• Sparsity oracle inequalities for the Lasso
• L1-Penalization for Mixture Regression Models
• Gaussian approximation of suprema of empirical processes
• Aggregation for Gaussian regression
• High-dimensional instrumental variables regression and confidence sets
• Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators
• Inference for High-Dimensional Sparse Econometric Models
• LASSO Methods for Gaussian Instrumental Variables Models
• Confidence Intervals for Low-Dimensional Parameters in High-Dimensional Linear Models

Cited by in corpus (9)

• Uniform Post Selection Inference for LAD Regression and Other Z-estimation problems
• On the Prediction Performance of the Lasso
• DoubleML -- An Object-Oriented Implementation of Double Machine Learning in R
• Variance prior forms for high-dimensional Bayesian variable selection
• Minimal penalties and the slope heuristics: a survey
• Optimal bounds for aggregation of affine estimators
• On estimation of the diagonal elements of a sparse precision matrix
• Pivotal estimation in high-dimensional regression via linear programming
• High-Dimensional Semiparametric Selection Models: Estimation Theory with an Application to the Retail Gasoline Market