A Rigorous Uncertainty-Aware Quantification Framework Is Essential for Reproducible and Replicable Machine Learning Workflows

Abstract

The ability to replicate predictions by machine learning (ML) or artificial intelligence (AI) models and results in scientific workflows that incorporate such ML/AI predictions is driven by numerous factors. An uncertainty-aware metric that can quantitatively assess the reproducibility of quantities of interest (QoI) would contribute to the trustworthiness of results obtained from scientific workflows involving ML/AI models. In this article, we discuss how uncertainty quantification (UQ) in a Bayesian paradigm can provide a general and rigorous framework for quantifying reproducibility for complex scientific workflows. Such as framework has the potential to fill a critical gap that currently exists in ML/AI for scientific workflows, as it will enable researchers to determine the impact of ML/AI model prediction variability on the predictive outcomes of ML/AI-powered workflows. We expect that the envisioned framework will contribute to the design of more reproducible and trustworthy workflows for diverse scientific applications, and ultimately, accelerate scientific discoveries.

References in corpus (11)

• ADADELTA: An Adaptive Learning Rate Method
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Interpretable machine learning: definitions, methods, and applications
• The importance of transparency and reproducibility in artificial intelligence research
• Ten Simple Rules for Making Research Software More Robust
• Autonomous Efficient Experiment Design for Materials Discovery with Bayesian Model Averaging
• Sources of Irreproducibility in Machine Learning: A Review
• When we can trust computers (and when we can't)
• Quantifying the multi-objective cost of uncertainty
• Reproducing GW150914: the first observation of gravitational waves from a binary black hole merger
• Optimal Experimental Design for Uncertain Systems Based on Coupled Differential Equations