Not Normal: the uncertainties of scientific measurements
arXiv:1612.00778 · doi:10.1098/rsos.160600
Abstract
Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how well these have been evaluated and what they imply. Reported scientific uncertainties were studied by analysing 41000 measurements of 3200 quantities from medicine, nuclear and particle physics, and interlaboratory comparisons ranging from chemistry to toxicology. Outliers are common, with 5σ disagreements up to five orders of magnitude more frequent than naively expected. Uncertainty-normalized differences between multiple measurements of the same quantity are consistent with heavy-tailed Student-t distributions that are often almost Cauchy, far from a Gaussian Normal bell curve. Medical research uncertainties are generally as well evaluated as those in physics, but physics uncertainty improves more rapidly, making feasible simple significance criteria such as the 5σ discovery convention in particle physics. Contributions to measurement uncertainty from mistakes and unknown problems are not completely unpredictable. Such errors appear to have power-law distributions consistent with how designed complex systems fail, and how unknown systematic errors are constrained by researchers. This better understanding may help improve analysis and meta-analysis of data, and help scientists and the public have more realistic expectations of what scientific results imply.
arXiv comment17 pages, 5 figures. Auxiliary Excel file (UncertaintyDataDescription.xls) lists sources of data
References in corpus (18)
- Power-law distributions in empirical data
- Big Bang Nucleosynthesis (in "The Review of Particle Properties" 2004)
- CODATA Recommended Values of the Fundamental Physical Constants: 2010
- BICEP2 I: Detection Of B-mode Polarization at Degree Angular Scales
- An investigation of the false discovery rate and the misinterpretation of P values
- Highly Optimized Tolerance: Robustness and Power Laws in Complex Systems
- Measurement of the neutrino velocity with the OPERA detector in the CNGS beam
- Fluctuation scaling in complex systems: Taylor's law and beyond
- The Proton Radius Puzzle
- Median Statistics, H_0, and the Accelerating Universe
- Non-Gaussian Error Distribution of Hubble Constant Measurements
- Optimal design, robustness, and risk aversion
- Non-Gaussian Error Distributions of LMC Distance Moduli Measurements
- Non-Gaussian Error Distribution of Abundance Measurements
- Definition and Treatment of Systematic Uncertainties in High Energy Physics and Astrophysics
- Evidence for Conservatism in LHC SUSY Searches
- On the probability distribution of the experimental results
- Lessons about likelihood functions from nuclear physics
Cited by in corpus (16)
- Median Statistics Estimate of the Distance to the Galactic Center
- The Modeling Framework for Experimental Physics: Description, development, and applications
- Bell correlations in a many-body system with finite statistics
- Reproducibility and Replication of Experimental Particle Physics Results
- Median Statistics Estimate of the Galactic Rotational Velocity
- MFV approach to robust estimate of neutron lifetime
- A meta-analysis of neutron lifetime measurements
- Non-Gaussian Error Distributions of Galactic Rotation Speed Measurements
- Nonparametric Simulation Extrapolation for Measurement Error Models
- Gaussian Processes, Median Statistics, Milky Way Rotation Curves
- A meta-analysis of distance measurements to M87
- An approach to robust Bayesian regression in astronomy
- Median Statistics Estimate of the Distance to M87
- Cosmological measurements from the CMB and BAO are insensitive to the tail probability in the assumed likelihood
- New Metric Formulas that Include Measurement Errors in Machine Learning for Natural Sciences
- Physics to the rescue?