Robustness and accuracy of methods for high dimensional data analysis based on Student's t statistic

Abstract

Student's $t$ statistic is finding applications today that were never envisaged when it was introduced more than a century ago. Many of these applications rely on properties, for example robustness against heavy tailed sampling distributions, that were not explicitly considered until relatively recently. In this paper we explore these features of the $t$ statistic in the context of its application to very high dimensional problems, including feature selection and ranking, highly multiple hypothesis testing, and sparse, high dimensional signal detection. Robustness properties of the $t$-ratio are highlighted, and it is established that those properties are preserved under applications of the bootstrap. In particular, bootstrap methods correct for skewness, and therefore lead to second-order accuracy, even in the extreme tails. Indeed, it is shown that the bootstrap, and also the more popular but less accurate $t$-distribution and normal approximations, are more effective in the tails than towards the middle of the distribution. These properties motivate new methods, for example bootstrap-based techniques for signal detection, that confine attention to the significant tail of a statistic.

37 pages, 5 figures

Cited by in corpus (5)

• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence
• Cramér-type moderate deviations for Studentized two-sample $U$-statistics with applications
• Unification of Rare and Weak Detection Models using Moderate Deviations Analysis and Log-Chisquared P-values
• An Extreme-Value Approach for Testing the Equality of Large U-Statistic Based Correlation Matrices