The long road to calibrated prediction uncertainty in computational chemistry

Abstract

Uncertainty quantification (UQ) in computational chemistry (CC) is still in its infancy. Very few CC methods are designed to provide a confidence level on their predictions, and most users still rely improperly on the mean absolute error as an accuracy metric. The development of reliable uncertainty quantification methods is essential, notably for computational chemistry to be used confidently in industrial processes. A review of the CC-UQ literature shows that there is no common standard procedure to report nor validate prediction uncertainty. I consider here analysis tools using concepts (calibration and sharpness) developed in meteorology and machine learning for the validation of probabilistic forecasters. These tools are adapted to CC-UQ and applied to datasets of prediction uncertainties provided by composite methods, Bayesian Ensembles methods, machine learning and a posteriori statistical methods.

References in corpus (28)

• Big Data meets Quantum Chemistry Approximations: The $\Delta$-Machine Learning Approach
• Machine learning prediction errors better than DFT accuracy
• Machine Learning of Molecular Electronic Properties in Chemical Compound Space
• Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• A Survey of Uncertainty in Deep Neural Networks
• Bayesian Error Estimation in Density Functional Theory
• $\Delta$-Machine Learning for Potential Energy Surfaces: A PIP approach to bring a DFT-based PES to CCSD(T) Level of Theory
• Uncertainty Quantification in Classical Molecular Dynamics
• Systematic Error Estimation for Chemical Reaction Energies
• A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
• Reliable estimation of prediction uncertainty for physico-chemical property models
• Gaussian Process-Based Refinement of Dispersion Corrections
• Putting Density Functional Theory to the Test in Machine-Learning-Accelerated Materials Discovery
• The parameters uncertainty inflation fallacy
• Probabilistic performance estimators for computational chemistry methods: the empirical cumulative distribution function of absolute errors
• Quantifying Confidence in Density Functional Theory Predicted Magnetic Ground States
• Impact of non-normal error distributions on the benchmarking and ranking of Quantum Machine Learning models
• Uncertainty Quantification of DFT-predicted Finite Temperature Thermodynamic Properties within the Debye Model
• Critical Benchmarking of the G4(MP2) Model, the Correlation Consistent Composite Approach and Popular Density Functional Approximations on a Probabilistically Pruned Benchmark Dataset of Formation Enthalpies
• Uncertainty Quantification in First-Principles Predictions of Harmonic Vibrational Frequencies of Molecules and Molecular Complexes
• A posteriori error estimation for the non-self-consistent Kohn-Sham equations
• Using the Gini coefficient to characterize the shape of computational chemistry error distributions
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• From Prescriptive to Predictive: an Interdisciplinary Perspective on the Future of Computational Chemistry
• Revisiting "What Every Computer Scientist Should Know About Floating-point Arithmetic"
• Verificarlo: checking floating point accuracy through Monte Carlo Arithmetic

Cited by in corpus (4)

• Universal QM/MM Approaches for General Nanoscale Applications
• Uncertainty-biased molecular dynamics for learning uniformly accurate interatomic potentials
• Prediction uncertainty validation for computational chemists
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity