Reproducibility, Replicability, and Repeatability: A survey of reproducible research with a focus on high performance computing

Abstract

Reproducibility is widely acknowledged as a fundamental principle in scientific research. Currently, the scientific community grapples with numerous challenges associated with reproducibility, often referred to as the ''reproducibility crisis.'' This crisis permeated numerous scientific disciplines. In this study, we examined the factors in scientific practices that might contribute to this lack of reproducibility. Significant focus is placed on the prevalent integration of computation in research, which can sometimes function as a black box in published papers. Our study primarily focuses on highperformance computing (HPC), which presents unique reproducibility challenges. This paper provides a comprehensive review of these concerns and potential solutions. Furthermore, we discuss the critical role of reproducible research in advancing science and identifying persisting issues within the field of HPC.

References in corpus (14)

• Can parametric statistical methods be trusted for fMRI based group studies?
• Measuring reproducibility of high-throughput experiments
• An introduction to Docker for reproducible research, with examples from the R environment
• Best Practices for Scientific Computing
• Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
• Sustainable computational science: the ReScience initiative
• Silent Data Corruptions at Scale
• Silent error detection in numerical time-stepping schemes
• Containerisation for High Performance Computing Systems: Survey and Prospects
• Containers for portable, productive and performant scientific computing
• Characterizing the Reproducibility of Noisy Quantum Circuits
• Numerical Reproducibility and Parallel Computations: Issues for Interval Algorithms
• Stability of noisy quantum computing devices
• A Survey on Reproducibility in Parallel Computing