Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings

Abstract

While traditional multiple testing procedures prohibit adaptive analysis choices made by users, Goeman and Solari (2011) proposed a simultaneous inference framework that allows users such flexibility while preserving high-probability bounds on the false discovery proportion (FDP) of the chosen set. In this paper, we propose a new class of such simultaneous FDP bounds, tailored for nested sequences of rejection sets. While most existing simultaneous FDP bounds are based on closed testing using global null tests based on sorted p-values, we additionally consider the setting where side information can be leveraged to boost power, the variable selection setting where knockoff statistics can be used to order variables, and the online setting where decisions about rejections must be made as data arrives. Our finite-sample, closed form bounds are based on repurposing the FDP estimates from false discovery rate (FDR) controlling procedures designed for each of the above settings. These results establish a novel connection between the parallel literatures of simultaneous FDP bounds and FDR control methods, and use proof techniques employing martingales and filtrations that are new to both these literatures. We demonstrate the utility of our results by augmenting a recent knockoffs analysis of the UK Biobank dataset.

References in corpus (15)

• Valid post-selection inference
• Controlling the false discovery rate via knockoffs
• Generalizations of the Familywise Error Rate
• A stochastic process approach to false discovery control
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• Multiple Testing for Exploratory Research
• Gene Hunting with Knockoffs for Hidden Markov Models
• An adaptive step-down procedure with proven FDR control under independence
• Post hoc inference via joint family-wise error rate control
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• STAR: A general interactive framework for FDR control under structural constraints
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• SAFFRON: an adaptive algorithm for online control of the false discovery rate
• Online control of the false discovery rate with decaying memory
• Differentially Private False Discovery Rate Control

Cited by in corpus (2)

• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Post hoc false discovery proportion inference under a Hidden Markov Model