Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. I. Theory

Abstract

The comparison of benchmark error sets is an essential tool for the evaluation of theories in computational chemistry. The standard ranking of methods by their Mean Unsigned Error is unsatisfactory for several reasons linked to the non-normality of the error distributions and the presence of underlying trends. Complementary statistics have recently been proposed to palliate such deficiencies, such as quantiles of the absolute errors distribution or the mean prediction uncertainty. We introduce here a new score, the systematic improvement probability (SIP), based on the direct system-wise comparison of absolute errors. Independently of the chosen scoring rule, the uncertainty of the statistics due to the incompleteness of the benchmark data sets is also generally overlooked. However, this uncertainty is essential to appreciate the robustness of rankings. In the present article, we develop two indicators based on robust statistics to address this problem: P_{inv}, the inversion probability between two values of a statistic, and \mathbf{P}_{r}, the ranking probability matrix. We demonstrate also the essential contribution of the correlations between error sets in these scores comparisons.

References in corpus (9)

• Big Data meets Quantum Chemistry Approximations: The $\Delta$-Machine Learning Approach
• Error estimates for solid-state density-functional theory predictions: an overview by means of the ground-state elemental crystals
• Bayesian Error Estimation in Density Functional Theory
• Reliable estimation of prediction uncertainty for physico-chemical property models
• Choosing a density functional for static molecular polarizabilities
• The parameters uncertainty inflation fallacy
• Probabilistic performance estimators for computational chemistry methods: the empirical cumulative distribution function of absolute errors
• Using the bootstrap to quantify the authority of an empirical ranking
• Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. II. Applications

Cited by in corpus (7)

• Uncertainty Quantification in Classical Molecular Dynamics
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
• Molecule-specific Uncertainty Quantification in Quantum Chemical Studies
• Impact of non-normal error distributions on the benchmarking and ranking of Quantum Machine Learning models
• Critical Benchmarking of the G4(MP2) Model, the Correlation Consistent Composite Approach and Popular Density Functional Approximations on a Probabilistically Pruned Benchmark Dataset of Formation Enthalpies
• Using the Gini coefficient to characterize the shape of computational chemistry error distributions
• Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. II. Applications