Why do some probabilistic forecasts lack reliability?

Abstract

In this work, we investigate the reliability of the probabilistic binary forecast. We mathematically prove that a necessary, but not sufficient, condition for achieving a reliable probabilistic forecast is maximizing the Peirce skill score (PSS) at the threshold probability of the climatological base rate. The condition is confirmed by using artificially synthesized forecast-outcome pair data and previously published probabilistic solar flare forecast models. The condition gives a partial answer as to why some probabilistic forecast system lack reliability, because the system, which does not satisfy the proved condition, can never be reliable. Therefore, the proved condition is very important for the developers of a probabilistic forecast system. The result implies that those who want to develop a reliable probabilistic forecast system must adjust or train the system so as to maximize PSS near the threshold probability of the climatological base rate.

arXiv comment12 pages, 6 figures, 1 table, accepted for publication in the Journal of Space Weather and Space Climate (JSWSC)

References in corpus (5)

• Solar Flare Prediction Model with Three Machine-Learning Algorithms Using Ultraviolet Brightening and Vector Magnetogram
• A Statistical Solar Flare Forecast Method
• UFCORIN: A Fully Automated Predictor of Solar Flares in GOES X-Ray Flux
• Flaring Rates and the Evolution of Sunspot Group McIntosh Classifications
• Verification of operational solar flare forecast: Case of Regional Warning Center Japan