Second order Stein: SURE for SURE and other applications in high-dimensional inference

Abstract

Stein's formula states that a random variable of the form $z^{\top }f(z)-\text{div}f(z)$ is mean-zero for functions $f$ with integrable gradient. Here, $\text{div}f$ is the divergence of the function $f$ and $z$ is a standard normal vector. This paper aims to propose a Second Order Stein formula to characterize the variance of such random variables for all functions $f(z)$ with square integrable gradient, and to demonstrate the usefulness of this formula in various applications. In the Gaussian sequence model, a consequence of Stein's formula is Stein's Unbiased Risk Estimate (SURE), an unbiased estimate of the mean squared risk for almost any estimator $\hat{\mu }$ of the unknown mean. A first application of the Second Order Stein formula is an Unbiased Risk Estimate for SURE itself (SURE for SURE): an unbiased estimate {providing} information about the squared distance between SURE and the squared estimation error of $\hat{\mu }$. SURE for SURE has a simple form as a function of the data and is applicable to all $\hat{\mu }$ with square integrable gradient, e.g. the Lasso and the Elastic Net. In addition to SURE for SURE, the following applications are developed: (1) Upper bounds on the risk of SURE when the estimation target is the mean squared error; (2) Confidence regions based on SURE; (3) Oracle inequalities satisfied by SURE-tuned estimates; (4) An upper bound on the variance of the size of the model selected by the Lasso; (5) Explicit expressions of SURE for SURE for the Lasso and the Elastic-Net; (6) In the linear model, a general semi-parametric scheme to de-bias a differentiable initial estimator for inference of a low-dimensional projection of the unknown $\beta$, with a characterization of the variance after de-biasing; and (7) An accuracy analysis of a Gaussian Monte Carlo scheme to approximate the divergence of functions $f:R^n\to R^n$.

References in corpus (8)

• Nearly unbiased variable selection under minimax concave penalty
• Simultaneous analysis of Lasso and Dantzig selector
• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Confidence sets in sparse regression
• Deviation optimal learning using greedy Q-aggregation
• A short survey of Stein's method
• De-Biasing The Lasso With Degrees-of-Freedom Adjustment
• The noise barrier and the large signal bias of the Lasso and other convex estimators

Cited by in corpus (5)

• The Lasso with general Gaussian designs with applications to hypothesis testing
• De-Biasing The Lasso With Degrees-of-Freedom Adjustment
• Out-of-sample error estimate for robust M-estimators with convex penalty
• Chi-square and normal inference in high-dimensional multi-task regression
• Asymptotic normality of robust $M$-estimators with convex penalty