An investigation of the false discovery rate and the misinterpretation of P values

Abstract

The following proposition is justified from several different points of view. If you use P = 0.05 to suggest that you have made a discovery, you will be wrong at least 30 percent of the time. If, as is often the case, experiments are under-powered, you will be wrong most of the time. It is concluded that if you wish to keep your false discovery rate below 5 percent, you need to use a 3-sigma rule, or to insist on P value below 0.001. And never use the word "significant".

arXiv commentSmall typo fixed again, at top of page 7 20 Nov

Cited by in corpus (25)

• Balancing Type I Error and Power in Linear Mixed Models
• The false positive risk: a proposal concerning what to do about p values
• P-values: misunderstood and misused
• Not Normal: the uncertainties of scientific measurements
• P-value: A Bless or A Curse for Evidence-Based Studies?
• Replication studies considered harmful
• Progenitor Mass Distribution for 22 Historic Core-Collapse Supernovae
• A search for transit timing variations within the exomoon corridor using Kepler data
• Embodied negation and levels of concreteness: A TMS Study on German and Italian language processing
• Replacing P values with frequentist posterior probabilities - as possible parameter values must have uniform base-rate prior probabilities by definition in a random sampling model
• A reckless guide to P-values: local evidence, global errors
• Experiments as Code: A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable Experiments
• Generative Adversarial Networks for geometric surfaces prediction in injection molding
• The impact of using biased performance metrics on software defect prediction research
• Active Invariant Causal Prediction: Experiment Selection through Stability
• Sensitivity Analysis of calibration methods and factors effecting the statistical nature of radiation measurement
• Discussion Paper: Should statistics rescue mathematical modelling?
• A reproducible effect size is more useful than an irreproducible hypothesis test to analyze high throughput sequencing datasets
• A response to critiques of "The reproducibility of research and the misinterpretation of p-values"
• Is neuroscience facing up to statistical power?
• Quantifying Local Randomness in Human DNA and RNA Sequences Using Erdos Motifs
• Dataset Optimization Strategies for MalwareTraffic Detection
• Generalised Linear Models for Dependent Binary Outcomes with Applications to Household Stratified Pandemic Influenza Data
• Characterizing interdisciplinarity in drug research: a translational science perspective
• Why Most Results of Socio-Technical Security User Studies Are False