Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression
arXiv:1804.02605 · doi:10.1093/imaiai/iaac012
Abstract
Concentration inequalities form an essential toolkit in the study of high dimensional (HD) statistical methods. Most of the relevant statistics literature in this regard is based on sub-Gaussian or sub-exponential tail assumptions. In this paper, we first bring together various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-Weibull) tail assumptions. These results extract a part sub-Gaussian tail behavior in finite samples, matching the asymptotics governed by the central limit theorem, and are compactly represented in terms of a new Orlicz quasi-norm - the Generalized Bernstein-Orlicz norm - that typifies such tail behaviors. We illustrate the usefulness of these inequalities through the analysis of four fundamental problems in HD statistics. In the first two problems, we study the rate of convergence of the sample covariance matrix in terms of the maximum elementwise norm and the maximum k-sub-matrix operator norm which are key quantities of interest in bootstrap, HD covariance matrix estimation and HD inference. The third example concerns the restricted eigenvalue condition, required in HD linear regression, which we verify for all sub-Weibull random vectors through a unified analysis, and also prove a more general result related to restricted strong convexity in the process. In the final example, we consider the Lasso estimator for linear regression and establish its rate of convergence under much weaker than usual tail assumptions (on the errors as well as the covariates), while also allowing for misspecified models and both fixed and random design. To our knowledge, these are the first such results for Lasso obtained in this generality. The common feature in all our results over all the examples is that the convergence rates under most exponential tails match the usual ones under sub-Gaussian assumptions.
arXiv comment68 pages; Revised version; To appear in Information and Inference: A Journal of the IMA
References in corpus (21)
- Simultaneous analysis of Lasso and Dantzig selector
- Covariance regularization by thresholding
- On the conditions used to prove oracle results for the Lasso
- Least squares after model selection in high-dimensional sparse models
- Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
- High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity
- Uniform in bandwidth consistency of kernel-type function estimators
- Concentration around the mean for maxima of empirical processes
- Gaussian approximation of suprema of empirical processes
- Exponential and moment inequalities for U-statistics
- Adaptive covariance matrix estimation through block thresholding
- Concentration inequalities and asymptotic results for ratio type empirical processes
- Moment inequalities for functions of independent random variables
- Nonparametric regression with nonparametrically generated covariates
- The lower tail of random quadratic forms, with applications to ordinary least squares and restricted eigenvalue properties
- Tail behaviour of multiple random integrals and U-statistics
- New concentration inequalities for suprema of empirical processes
- Moment estimates for chaoses generated by symmetric random variables with logarithmically convex tails
- Reluctant Interaction Modeling
- Suprema of canonical Weibull processes
- Gaussian martingale inequality applies to random functions and maxima of empirical processes
Cited by in corpus (22)
- Generalized Dynamic Factor Models and Volatilities: Consistency, rates, and prediction intervals
- Sharp Convergence Rates for Empirical Optimal Transport with Smooth Costs
- Bootstrapping Upper Confidence Bound
- Next Generation Models for Portfolio Risk Management: An Approach Using Financial Big Data
- Sharper Sub-Weibull Concentrations
- Reluctant Interaction Modeling
- Double Robust Semi-Supervised Inference for the Mean: Selection Bias under MAR Labeling with Decaying Overlap
- Variational Inference in high-dimensional linear regression
- High Dimensional M-Estimation with Missing Outcomes: A Semi-Parametric Framework
- Hypothesis testing for eigenspaces of covariance matrix
- Large-Sample Properties of Blind Estimation of the Linear Discriminant Using Projection Pursuit
- On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning
- Semiparametric Efficiency in Convexity Constrained Single Index Model
- A Stochastic Operator Framework for Optimization and Learning with Sub-Weibull Errors
- High-dimensional Functional Graphical Model Structure Learning via Neighborhood Selection Approach
- Bayesian neural network unit priors and generalized Weibull-tail property
- Outlier-Robust Learning of Ising Models Under Dobrushin's Condition
- Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
- Robust Inference for High-Dimensional Linear Models via Residual Randomization
- Sparse Regression for Extreme Values
- All of Linear Regression
- Uniform-in-Submodel Bounds for Linear Regression in a Model Free Framework