Reliability, Sufficiency, and the Decomposition of Proper Scores

Abstract

Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. In the binary case, scoring rules (which are strictly proper) allow for a decomposition into terms related to the resolution and to the reliability of the forecast. This fact is particularly well known for the Brier Score. In this paper, this result is extended to forecasts for finite--valued targets. Both resolution and reliability are shown to have a positive effect on the score. It is demonstrated that resolution and reliability are directly related to forecast attributes which are desirable on grounds independent of the notion of scores. This finding can be considered an epistemological justification of measuring forecast quality by proper scores. A link is provided to the original work of DeGroot et al (1982), extending their concepts of sufficiency and refinement. The relation to the conjectured sharpness principle of Gneiting et al (2005a) is elucidated.

v1: 9 pages; submitted to International Journal of Forecasting v2: 12 pages; Significant change of contents; stronger focus on decomposition; Extensive comments on and extensions of earlier work, in particular sufficiency

Cited by in corpus (19)

• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• A review of predictive uncertainty estimation with machine learning
• Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited
• Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination
• The role of the information set for forecasting - with applications to risk management
• Predictability of extreme events in social media
• Why ex post peer review encourages high-risk research while ex ante review discourages it
• Quantile forecast discrimination ability and value
• Variance estimation for Brier Score decomposition
• Assessing the reliability of ensemble forecasting systems under serial dependence
• Calibrating sufficiently
• Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation
• Slicing Through Bias: Explaining Performance Gaps in Medical Image Analysis using Slice Discovery Methods
• Recalibrating probabilistic forecasts of epidemics
• Probabilistic prediction of the time to hard freeze using seasonal weather forecasts and survival time methods
• Post-processing of ensemble photovoltaic power forecasts with distributional and quantile regression methods
• h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective
• Post-processing of wind gusts from COSMO-REA6 with a spatial Bayesian hierarchical extreme value model