Prediction uncertainty validation for computational chemists
arXiv:2204.13477 · doi:10.1063/5.0109572
Abstract
Validation of prediction uncertainty (PU) is becoming an essential task for modern computational chemistry. Designed to quantify the reliability of predictions in meteorology, the calibration-sharpness (CS) framework is now widely used to optimize and validate uncertainty-aware machine learning (ML) methods. However, its application is not limited to ML and it can serve as a principled framework for any PU validation. The present article is intended as a step-by-step introduction to the concepts and techniques of PU validation in the CS framework, adapted to the specifics of computational chemistry. The presented methods range from elementary graphical checks to more sophisticated ones based on local calibration statistics. The concept of tightness, is introduced. The methods are illustrated on synthetic datasets and applied to uncertainty quantification data extracted from the computational chemistry literature.
arXiv commentAccepted for publication in J. Chem. Phys
References in corpus (24)
- Scikit-learn: Machine Learning in Python
- Less is more: sampling chemical space with active learning
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- Fast and Accurate Uncertainty Estimation in Chemical Machine Learning
- Uncertainty estimation for molecular dynamics and sampling
- Systematic Error Estimation for Chemical Reaction Energies
- A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy
- Reliable estimation of prediction uncertainty for physico-chemical property models
- Gaussian Process-Based Refinement of Dispersion Corrections
- Calibration for the (Computationally-Identifiable) Masses
- The parameters uncertainty inflation fallacy
- Probabilistic performance estimators for computational chemistry methods: the empirical cumulative distribution function of absolute errors
- Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification
- Molecule-specific Uncertainty Quantification in Quantum Chemical Studies
- The long road to calibrated prediction uncertainty in computational chemistry
- Impact of non-normal error distributions on the benchmarking and ranking of Quantum Machine Learning models
- Evaluating Forecasts with scoringutils in R
- Uncertainty Quantification in First-Principles Predictions of Harmonic Vibrational Frequencies of Molecules and Molecular Complexes
- Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
- Using the Gini coefficient to characterize the shape of computational chemistry error distributions
- Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection
- Confidence curves for UQ validation: probabilistic reference vs. oracle
- Heuristics and Uncertainty Quantification in Rational and Inverse Compound and Catalyst Design
- Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials