Asynchronous Online Testing of Multiple Hypotheses

Abstract

We consider the problem of asynchronous online testing, aimed at providing control of the false discovery rate (FDR) during a continual stream of data collection and testing, where each test may be a sequential test that can start and stop at arbitrary times. This setting increasingly characterizes real-world applications in science and industry, where teams of researchers across large organizations may conduct tests of hypotheses in a decentralized manner. The overlap in time and space also tends to induce dependencies among test statistics, a challenge for classical methodology, which either assumes (overly optimistically) independence or (overly pessimistically) arbitrary dependence between test statistics. We present a general framework that addresses both of these issues via a unified computational abstraction that we refer to as "conflict sets." We show how this framework yields algorithms with formal FDR guarantees under a more intermediate, local notion of dependence. We illustrate our algorithms in simulations by comparing to existing algorithms for online FDR control.

arXiv comment36 pages, 16 figures

References in corpus (6)

• Active sequential hypothesis testing
• The Ladder: A Reliable Leaderboard for Machine Learning Competitions
• SAFFRON: an adaptive algorithm for online control of the false discovery rate
• Power of Ordered Hypothesis Testing
• A Bandit Approach to Multiple Testing with False Discovery Control
• Online control of the false discovery rate in biomedical research

Cited by in corpus (5)

• ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls
• Online Control of the False Coverage Rate and False Sign Rate
• Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep
• Online control of the familywise error rate
• Asymptotically optimal sequential FDR and pFDR control with (or without) prior information on the number of signals