A More Powerful Two-Sample Test in High Dimensions using Random Projection

Abstract

We consider the hypothesis testing problem of detecting a shift between the means of two multivariate normal distributions in the high-dimensional setting, allowing for the data dimension p to exceed the sample size n. Specifically, we propose a new test statistic for the two-sample test of means that integrates a random projection with the classical Hotelling T^2 statistic. Working under a high-dimensional framework with (p,n) tending to infinity, we first derive an asymptotic power function for our test, and then provide sufficient conditions for it to achieve greater power than other state-of-the-art tests. Using ROC curves generated from synthetic data, we demonstrate superior performance against competing tests in the parameter regimes anticipated by our theoretical results. Lastly, we illustrate an advantage of our procedure's false positive rate with comparisons on high-dimensional gene expression data involving the discrimination of different types of cancer.

Version 3 is an extended version of our NIPS 2011 conference paper. This should be regarded as the final version and cited as a NIPS 2011 paper. Note that version3=version1. Also, version 2 should be considered as defunct, as it contains an error in the variance formula in equation (4)

References in corpus (3)

• A two-sample test for high-dimensional data with applications to gene-set testing
• A Bernstein-type inequality for stochastic processes of quadratic forms of Gaussian variables
• Testing for Homogeneity with Kernel Fisher Discriminant Analysis

Cited by in corpus (14)

• Gains in Power from Structured Two-Sample Tests of Means on Graphs
• More power via graph-structured tests for differential expression of gene networks
• A high-dimensional two-sample test for the mean using random subspaces
• An adaptable generalization of Hotelling's $T^2$ test in high dimension
• A Kernel Two-Sample Test for Functional Data
• Estimating Unknown Sparsity in Compressed Sensing
• Multiple-Splitting Projection Test for High-Dimensional Mean Vectors
• On the High-dimensional Power of Linear-time Kernel Two-Sample Testing under Mean-difference Alternatives
• Supervised Dimensionality Reduction for Big Data
• Online landmark replacement for out-of-sample dimensionality reduction methods
• Randomized tests for high-dimensional regression: A more efficient and powerful solution
• Mean Test with Fewer Observation than Dimension and Ratio Unbiased Estimator for Correlation Matrix
• Sparse sketches with small inversion bias
• Learning Augmentation Distributions using Transformed Risk Minimization