QuickMMCTest - Quick Multiple Monte Carlo Testing

Abstract

Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p-values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.

References in corpus (3)

• Analysis of Thompson Sampling for the multi-armed bandit problem
• Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn
• MMCTest - A Safe Algorithm for Implementing Multiple Monte Carlo Tests

Cited by in corpus (3)

• Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits
• Implementing Monte Carlo Tests with P-value Buckets
• Conformal novelty detection for replicate point patterns with FDR or FWER control