Semi-supervised multiple testing

Abstract

An important limitation of standard multiple testing procedures is that the null distribution should be known. Here, we consider a null distribution-free approach for multiple testing in the following semi-supervised setting: the user does not know the null distribution, but has at hand a sample drawn from this null distribution. In practical situations, this null training sample (NTS) can come from previous experiments, from a part of the data under test, from specific simulations, or from a sampling process. In this work, we present theoretical results that handle such a framework, with a focus on the false discovery rate (FDR) control and the Benjamini-Hochberg (BH) procedure. First, we provide upper and lower bounds for the FDR of the BH procedure based on empirical $p$-values. These bounds match when $\alpha (n+1)/m$ is an integer, where $n$ is the NTS sample size and $m$ is the number of tests. Second, we give a power analysis for that procedure suggesting that the price to pay for ignoring the null distribution is low when $n$ is sufficiently large in front of $m$; namely $n\gtrsim m/(\max (1,k))$, where $k$ denotes the number of ``detectable'' alternatives. Third, to complete the picture, we also present a negative result that evidences an intrinsic transition phase to the general semi-supervised multiple testing problem {and shows that the empirical BH method is optimal in the sense that its performance boundary follows this transition phase}. Our theoretical properties are supported by numerical experiments, which also show that the delineated boundary is of correct order without further tuning any constant. Finally, we demonstrate that our work provides a theoretical ground for standard practice in astronomical data analysis, and in particular for the procedure proposed in \cite{Origin2020} for galaxy detection.

References in corpus (12)

• Auto-Encoding Variational Bayes
• Control of generalized error rates in multiple testing
• Testing for Outliers with Conformal p-values
• Replicability analysis for genome-wide association studies
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• A study of periodograms standardized using training data sets and application to exoplanet detection
• Auto-Encoding Knockoff Generator for FDR Controlled Variable Selection
• Testing over a continuum of null hypotheses with False Discovery Rate control
• Multiple Testing in Nonparametric Hidden Markov Models: An Empirical Bayes Approach
• Solving the Empirical Bayes Normal Means Problem with Correlated Noise
• Dynamic Algorithms for Online Multiple Testing
• Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits

Cited by in corpus (1)

• FDR Control for Online Anomaly Detection