Post hoc inference via joint family-wise error rate control

Abstract

We introduce a general methodology for post hoc inference in a large-scale multiple testing framework. The approach is called "user-agnostic" in the sense that the statistical guarantee on the number of correct rejections holds for any set of candidate items selected by the user (after having seen the data). This task is investigated by defining a suitable criterion, named the joint-family-wise-error rate (JER for short). We propose several procedures for controlling the JER, with a special focus on incorporating dependencies while adapting to the unknown quantity of signal (via a step-down approach). We show that our proposed setting incorporates as particular cases a version of the higher criticism as well as the closed testing based approach of Goeman and Solari (2011). Our theoretical statements are supported by numerical experiments.

References in corpus (2)

• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Post hoc false positive control for spatially structured hypotheses

Cited by in corpus (10)

• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Notip: Non-parametric True Discovery Proportion control for brain imaging
• Permutation-Based True Discovery Guarantee by Sum Tests
• Hierarchical correction of p-values via an ultrametric tree running Ornstein-Uhlenbeck process
• False Discovery Proportion control for aggregated Knockoffs
• Post hoc false discovery proportion inference under a Hidden Markov Model
• FDP control in multivariate linear models using the bootstrap
• Simultaneous directional inference
• On agnostic post hoc approaches to false positive control