Why scoring functions cannot assess tail properties

Abstract

Motivated by the growing interest in sound forecast evaluation techniques with an emphasis on distribution tails rather than average behaviour, we investigate a fundamental question arising in this context: Can statistical features of distribution tails be elicitable, i.e. be the unique minimizer of an expected score? We demonstrate that expected scores are not suitable to distinguish genuine tail properties in a very strong sense. Specifically, we introduce the class of max-functionals, which contains key characteristics from extreme value theory, for instance the extreme value index. We show that its members fail to be elicitable and that their elicitation complexity is in fact infinite under mild regularity assumptions. Further we prove that, even if the information of a max-functional is reported via the entire distribution function, a proper scoring rule cannot separate max-functional values. These findings highlight the caution needed in forecast evaluation and statistical inference if relevant information is encoded by such functionals.

arXiv comment18 pages

References in corpus (3)

• Higher order elicitability and Osband's principle
• Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction
• Order-Sensitivity and Equivariance of Scoring Functions

Cited by in corpus (8)

• A review of predictive uncertainty estimation with machine learning
• Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval
• Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions
• Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals
• Hydrological post-processing for predicting extreme quantiles
• Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation
• Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules
• A review of machine learning concepts and methods for addressing challenges in probabilistic hydrological post-processing and forecasting