Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

Abstract

This review aims to draw attention to two issues of concern when we set out to make machine learning work in the chemical and materials domain, i.e., statistical loss function metrics for the validation and benchmarking of data-derived models, and the uncertainty quantification of predictions made by them. They are often overlooked or underappreciated topics as chemists typically only have limited training in statistics. Aside from helping to assess the quality, reliability, and applicability of a given model, these metrics are also key to comparing the performance of different models and thus for developing guidelines and best practices for the successful application of machine learning in chemistry.

References in corpus (7)

• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
• The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling
• Fast and Accurate Uncertainty Estimation in Chemical Machine Learning
• Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Impact of non-normal error distributions on the benchmarking and ranking of Quantum Machine Learning models
• Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. I. Theory

Cited by in corpus (6)

• How to validate machine-learned interatomic potentials
• The long road to calibrated prediction uncertainty in computational chemistry
• Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet
• Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
• Building Robust Machine Learning Models for Small Chemical Science Data: The Case of Shear Viscosity
• Heuristics and Uncertainty Quantification in Rational and Inverse Compound and Catalyst Design