Adjusting for selection bias in testing multiple families of hypotheses

Abstract

In many large multiple testing problems the hypotheses are divided into families. Given the data, families with evidence for true discoveries are selected, and hypotheses within them are tested. Neither controlling the error-rate in each family separately nor controlling the error-rate over all hypotheses together can assure that an error-rate is controlled in the selected families. We formulate this concern about selective inference in its generality, for a very wide class of error-rates and for any selection criterion, and present an adjustment of the testing level inside the selected families that retains the average error-rate over the selected families.

Cited by in corpus (4)

• When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing
• Accumulation tests for FDR control in ordered hypothesis testing
• A Practitioner's Guide to Multiple Testing Error Rates
• Multiscale statistical testing for connectome-wide association studies in fMRI