A Practical Probabilistic Benchmark for AI Weather Models

Abstract

Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.

15 pages, 5 figures

References in corpus (10)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
• Neural General Circulation Models for Weather and Climate
• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• ClimaX: A foundation model for weather and climate
• Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for WeatherBench
• Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models
• Forecasting Global Weather with Graph Neural Networks
• Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere
• FuXi-ENS: A machine learning model for medium-range ensemble weather forecasting

Cited by in corpus (1)

• Community Research Earth Digital Intelligence Twin (CREDIT)