High Dimensional Robust M-Estimation: Asymptotic Variance via Approximate Message Passing

Abstract

In a recent article (Proc. Natl. Acad. Sci., 110(36), 14557-14562), El Karoui et al. study the distribution of robust regression estimators in the regime in which the number of parameters p is of the same order as the number of samples n. Using numerical simulations and `highly plausible' heuristic arguments, they unveil a striking new phenomenon. Namely, the regression coefficients contain an extra Gaussian noise component that is not explained by classical concepts such as the Fisher information matrix. We show here that that this phenomenon can be characterized rigorously techniques that were developed by the authors to analyze the Lasso estimator under high-dimensional asymptotics. We introduce an approximate message passing (AMP) algorithm to compute M-estimators and deploy state evolution to evaluate the operating characteristics of AMP and so also M-estimates. Our analysis clarifies that the `extra Gaussian noise' encountered in this problem is fundamentally similar to phenomena already studied for regularized least squares in the setting n<p.

32 pages, 5 figures (v2 contains numerical simulations)

References in corpus (1)

• Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learning

Cited by in corpus (11)

• Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results
• Precise Error Analysis of Regularized M-estimators in High-dimensions
• False Discoveries Occur Early on the Lasso Path
• Distributed linear regression by averaging
• Robust Covariance and Scatter Matrix Estimation under Huber's Contamination Model
• Variance Breakdown of Huber (M)-estimators: $n/p\to m\in (1,\infty )$
• Can we trust the bootstrap in high-dimension?
• SLOPE is Adaptive to Unknown Sparsity and Asymptotically Minimax
• Non-negative Principal Component Analysis: Message Passing Algorithms and Sharp Asymptotics
• The Complete Lasso Tradeoff Diagram
• Sparsest Error Detection via Sparsity Invariant Transformation based ${\ell }_1$ Minimization