An investigation of the false discovery rate and the misinterpretation of P values

Abstract

The following proposition is justified from several different points of view. If you use P = 0.05 to suggest that you have made a discovery, you will be wrong at least 30 percent of the time. If, as is often the case, experiments are under-powered, you will be wrong most of the time. It is concluded that if you wish to keep your false discovery rate below 5 percent, you need to use a 3-sigma rule, or to insist on P value below 0.001. And never use the word "significant".

Small typo fixed again, at top of page 7 20 Nov

Cited by in corpus (15)

• Not Normal: the uncertainties of scientific measurements
• Progenitor Mass Distribution for 22 Historic Core-Collapse Supernovae
• A search for transit timing variations within the exomoon corridor using Kepler data
• Embodied negation and levels of concreteness: A TMS Study on German and Italian language processing
• A reckless guide to P-values: local evidence, global errors
• Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments
• Generative Adversarial Networks for geometric surfaces prediction in injection molding
• Active Invariant Causal Prediction: Experiment Selection through Stability
• The impact of using biased performance metrics on software defect prediction research
• Sensitivity Analysis of calibration methods and factors effecting the statistical nature of radiation measurement
• Is neuroscience facing up to statistical power?
• Dataset Optimization Strategies for MalwareTraffic Detection
• Why Most Results of Socio-Technical Security User Studies Are False
• Generalised Linear Models for Dependent Binary Outcomes with Applications to Household Stratified Pandemic Influenza Data
• Characterizing interdisciplinarity in drug research: a translational science perspective