Reproducibility and Replication of Experimental Particle Physics Results

Abstract

Recently, much attention has been focused on the replicability of scientific results, causing scientists, statisticians, and journal editors to examine closely their methodologies and publishing criteria. Experimental particle physicists have been aware of the precursors of non-replicable research for many decades and have many safeguards to ensure that the published results are as reliable as possible. The experiments require large investments of time and effort to design, construct, and operate. Large collaborations produce and check the results, and many papers are signed by more than three thousand authors. This paper gives an introduction to what experimental particle physics is and to some of the tools that are used to analyze the data. It describes the procedures used to ensure that results can be computationally reproduced, both by collaborators and by non-collaborators. It describes the status of publicly available data sets and analysis tools that aid in reproduction and recasting of experimental results. It also describes methods particle physicists use to maximize the reliability of the results, which increases the probability that they can be replicated by other collaborations or even the same collaborations with more data and new personnel. Examples of results that were later found to be false are given, both with failed replication attempts and one with alarmingly successful replications. While some of the characteristics of particle physics experiments are unique, many of the procedures and techniques can be and are used in other fields.

arXiv comment50 pages, 6 figures. Please see https://hdsr.mitpress.mit.edu/pub/32yz0u49/release/1 for a thoughtful comment by Andrew Fowlie, and https://hdsr.mitpress.mit.edu/pub/57tywz64/release/1 for the authors' response

References in corpus (35)

• Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
• Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC
• Big Bang Nucleosynthesis (in "The Review of Particle Properties" 2004)
• GW151226: Observation of Gravitational Waves from a 22-Solar-Mass Binary Black Hole Coalescence
• A Unified Approach to the Classical Statistical Analysis of Small Signals
• Machine learning and the physical sciences
• Confidence Level Computation for Combining Searches with Small Statistics
• Evidence for Neutrino Oscillations from the Observation of Electron Anti-neutrinos in a Muon Anti-Neutrino Beam
• Observation of Top Quark Production in Pbar-P Collisions
• Observation of the Top Quark
• Observation of $J/\psi p$ resonances consistent with pentaquark states in ${\Lambda }_b^0\to J/\psi K^{-}p$ decays
• The T2K Experiment
• Global analysis of three-flavour neutrino oscillations: synergies and tensions in the determination of theta_23, delta_CP, and the mass ordering
• Event generator tunes obtained from underlying event and multiparton scattering measurements
• Rivet user manual
• Improved Search for ${\bar{\nu }}_{\mu }\to {\bar{\nu }}_e$ Oscillations in the MiniBooNE Experiment
• An improved upper limit on the neutrino mass from a direct kinematic method by KATRIN
• Significant Excess of ElectronLike Events in the MiniBooNE Short-Baseline Neutrino Experiment
• Deuterium target data for precision neutrino-nucleus cross sections
• Neutrino Mass Ordering from Oscillations and Beyond: 2018 Status and Future Prospects
• Search for electroweak production of supersymmetric particles in final states with two or three leptons at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for chargino-neutralino production with mass splittings near the electroweak scale in three-lepton final states in $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector
• Search for direct stau production in events with two hadronic $\tau$-leptons in $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• Search for direct production of electroweakinos in final states with one lepton, missing transverse momentum and a Higgs boson decaying into two $b$-jets in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Reinterpretation of LHC Results for New Physics: Status and Recommendations after Run 2
• Improved Constraints on Sterile Neutrino Mixing from Disappearance Searches in the MINOS, MINOS+, Daya Bay, and Bugey-3 Experiments
• Measurements of the Higgs boson inclusive and differential fiducial cross sections in the 4$\ell$ decay channel at $\sqrt{s}$ = 13 TeV
• Systematic Errors: facts and fictions
• Search for bottom-squark pair production with the ATLAS detector in final states containing Higgs bosons, $b$-jets and missing transverse momentum
• Not Normal: the uncertainties of scientific measurements
• Asymmetric Statistical Errors
• Development and validation of HERWIG 7 tunes from CMS underlying-event measurements
• Asymmetric Systematic Errors
• Search for supersymmetry in events with a photon, a lepton, and missing transverse momentum in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Pathological Science

Cited by in corpus (4)

• Simple and statistically sound recommendations for analysing physical theories
• A Systematic Literature Review of Undergraduate Data Science Education Research
• Nested sampling for frequentist computation: fast estimation of small $p$-values
• Comment on "Reproducibility and Replication of Experimental Particle Physics Results"