Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression

Abstract

Concentration inequalities form an essential toolkit in the study of high dimensional (HD) statistical methods. Most of the relevant statistics literature in this regard is based on sub-Gaussian or sub-exponential tail assumptions. In this paper, we first bring together various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-Weibull) tail assumptions. These results extract a part sub-Gaussian tail behavior in finite samples, matching the asymptotics governed by the central limit theorem, and are compactly represented in terms of a new Orlicz quasi-norm - the Generalized Bernstein-Orlicz norm - that typifies such tail behaviors. We illustrate the usefulness of these inequalities through the analysis of four fundamental problems in HD statistics. In the first two problems, we study the rate of convergence of the sample covariance matrix in terms of the maximum elementwise norm and the maximum k-sub-matrix operator norm which are key quantities of interest in bootstrap, HD covariance matrix estimation and HD inference. The third example concerns the restricted eigenvalue condition, required in HD linear regression, which we verify for all sub-Weibull random vectors through a unified analysis, and also prove a more general result related to restricted strong convexity in the process. In the final example, we consider the Lasso estimator for linear regression and establish its rate of convergence under much weaker than usual tail assumptions (on the errors as well as the covariates), while also allowing for misspecified models and both fixed and random design. To our knowledge, these are the first such results for Lasso obtained in this generality. The common feature in all our results over all the examples is that the convergence rates under most exponential tails match the usual ones under sub-Gaussian assumptions.

68 pages; Revised version; To appear in Information and Inference: A Journal of the IMA

References in corpus (7)

• Covariance regularization by thresholding
• Concentration around the mean for maxima of empirical processes
• Adaptive covariance matrix estimation through block thresholding
• Moment inequalities for functions of independent random variables
• Nonparametric regression with nonparametrically generated covariates
• Reluctant Interaction Modeling
• Gaussian martingale inequality applies to random functions and maxima of empirical processes

Cited by in corpus (21)

• Generalized Dynamic Factor Models and Volatilities: Consistency, rates, and prediction intervals
• Sharp Convergence Rates for Empirical Optimal Transport with Smooth Costs
• Bootstrapping Upper Confidence Bound
• Next Generation Models for Portfolio Risk Management: An Approach Using Financial Big Data
• Sharper Sub-Weibull Concentrations
• Reluctant Interaction Modeling
• Variational Inference in high-dimensional linear regression
• Double Robust Semi-Supervised Inference for the Mean: Selection Bias under MAR Labeling with Decaying Overlap
• Hypothesis testing for eigenspaces of covariance matrix
• High Dimensional M-Estimation with Missing Outcomes: A Semi-Parametric Framework
• Large-Sample Properties of Blind Estimation of the Linear Discriminant Using Projection Pursuit
• On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning
• A Stochastic Operator Framework for Optimization and Learning with Sub-Weibull Errors
• Outlier-Robust Learning of Ising Models Under Dobrushin's Condition
• High-dimensional Functional Graphical Model Structure Learning via Neighborhood Selection Approach
• Bayesian neural network unit priors and generalized Weibull-tail property
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
• Sparse Regression for Extreme Values
• Robust Inference for High-Dimensional Linear Models via Residual Randomization
• All of Linear Regression
• Uniform-in-Submodel Bounds for Linear Regression in a Model Free Framework