Variance Breakdown of Huber (M)-estimators: $n/p\to m\in (1,\infty )$

Abstract

A half century ago, Huber evaluated the minimax asymptotic variance in scalar location estimation, ${\min }_{\psi }{\max }_{F\in {\mathcal{F}}_{\epsilon }}V(\psi ,F)=\frac{1}{I(F_{\epsilon }^{\ast })}$, where $V(\psi ,F)$ denotes the asymptotic variance of the $(M)$-estimator for location with score function $\psi$, and $I(F_{\epsilon }^{\ast })$ is the minimal Fisher information ${\min }_{{\mathcal{F}}_{\epsilon }}I(F)$ over the class of $\epsilon$-Contaminated Normal distributions. We consider the linear regression model $Y=X{\theta }_0+W$, $W_i{\sim }_{\text{i.i.d.}}F$, and iid Normal predictors $X_{i,j}$, working in the high-dimensional-limit asymptotic where the number $n$ of observations and $p$ of variables both grow large, while $n/p\to m\in (1,\infty )$; hence $m$ plays the role of `asymptotic number of observations per parameter estimated'. Let $V_m(\psi ,F)$ denote the per-coordinate asymptotic variance of the $(M)$-estimator of regression in the $n/p\to m$ regime. Then $V_m\ne V$; however $V_m\to V$ as $m\to \infty$. In this paper we evaluate the minimax asymptotic variance of the Huber $(M)$-estimate. The statistician minimizes over the family $({\psi }_{\lambda }{)}_{\lambda >0}$ of all tunings of Huber $(M)$-estimates of regression, and Nature maximizes over gross-error contaminations $F\in {\mathcal{F}}_{\epsilon }$. Suppose that $I(F_{\epsilon }^{\ast })\cdot m>1$. Then ${\min }_{\lambda }{\max }_{F\in {\mathcal{F}}_{\epsilon }}{V}_m({\psi }_{\lambda },F)=\frac{1}{I(F_{\epsilon }^{\ast })-1/m}$. Strikingly, if $I(F_{\epsilon }^{\ast })\cdot m\le 1$, then the minimax asymptotic variance is $+\infty$. The breakdown point is where the Fisher information per parameter equals unity.

arXiv commentBased on a lecture delivered at a special colloquium honoring the 50th anniversary of the Seminar für Statistik (SfS) at ETH Zürich, November 25, 2014

References in corpus (3)

• Simultaneous analysis of Lasso and Dantzig selector
• Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results
• High Dimensional Robust M-Estimation: Asymptotic Variance via Approximate Message Passing

Cited by in corpus (7)

• Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions
• Precise Error Analysis of Regularized M-estimators in High-dimensions
• False Discoveries Occur Early on the Lasso Path
• Robust Covariance and Scatter Matrix Estimation under Huber's Contamination Model
• Robust Regression via Mutivariate Regression Depth
• Consistent Risk Estimation in Moderately High-Dimensional Linear Regression
• Low noise sensitivity analysis of Lq-minimization in oversampled systems