BONuS: Multiple multivariate testing with a data-adaptive test statistic
arXiv:2106.15743
Abstract
We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric. BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR). The BONuS procedure is closely connected to many existing FDR-controlling methods, such as the counting knockoffs procedure and Benjamini-Hochberg procedure with conformal p-values. Contrary to procedures that start with a -value for each hypothesis, BONuS analyzes the entire data set to adaptively estimate an optimal -value transform based on an empirical Bayes model. Moreover, it can iteratively refine the model fit by peeling off synthetic null statistics to reduce noise in a highly adaptively way. Our method controls FDR in finite samples even if the empirical Bayes model is incorrect or the estimation is poor. An extension, the Double BONuS procedure, validates the empirical Bayes model to guard against power loss due to model misspecification. We demonstrate the power gain of BONuS procedure on multi-trait GWAS, asset pricing, and LLM evaluation.
References in corpus (8)
- On the Benjamini--Hochberg method
- STAR: A general interactive framework for FDR control under structural constraints
- Multiple testing with the structure adaptive Benjamini-Hochberg algorithm
- A Power and Prediction Analysis for Knockoffs with Lasso Statistics
- A Generalized Benjamini-Hochberg Procedure for Multivariate Hypothesis Testing
- DAGGER: A sequential algorithm for FDR control on DAGs
- A New Approach for Large Scale Multiple Testing with Application to FDR Control for Graphically Structured Hypotheses
- Family learning: nonparametric statistical inference with parametric efficiency