Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events

Abstract

The forecast accuracy of machine learning (ML) weather prediction models is improving rapidly, leading many to speak of a "second revolution in weather forecasting". With numerous methods being developed and limited physical guarantees offered by ML models, there is a critical need for a comprehensive evaluation of these emerging techniques. While this need has been partly fulfilled by benchmark datasets, they provide little information on rare and impactful extreme events or on compound impact metrics, for which model accuracy might degrade due to misrepresented dependencies between variables. To address these issues, we compare ML weather prediction models (GraphCast, PanguWeather, and FourCastNet) and ECMWF's high-resolution forecast system (HRES) in three case studies: the 2021 Pacific Northwest heatwave, the 2023 South Asian humid heatwave, and the North American winter storm in 2021. We find that ML weather prediction models locally achieve similar accuracy to HRES on the record-shattering Pacific Northwest heatwave but underperform when aggregated over space and time. However, they forecast the compound winter storm substantially better. We also highlight structural differences in how the errors of HRES and the ML models build up to that event. The ML forecasts lack important variables for a detailed assessment of the health risks of the 2023 humid heatwave. Using a possible substitute variable, prediction errors show spatial patterns with the highest danger levels over Bangladesh being underestimated by the ML models. Generally, case-study-driven, impact-centric evaluation can complement existing research, increase public trust, and aid in developing reliable ML weather prediction models.

References in corpus (18)

• FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
• Neural General Circulation Models for Weather and Climate
• ClimaX: A foundation model for weather and climate
• The rise of data-driven weather forecasting
• AIFS -- ECMWF's data-driven forecasting system
• GenCast: Diffusion-based ensemble forecasting for medium-range weather
• Deep Learning for Day Forecasts from Sparse Observations
• WeatherBench 2: A benchmark for the next generation of data-driven global weather models
• Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification
• Scaling transformer neural networks for skillful and reliable medium-range weather forecasting
• FuXi: A cascade machine learning forecasting system for 15-day global weather forecast
• Deep graphical regression for jointly moderate and extreme Australian wildfires
• Regression modelling of spatiotemporal extreme U.S. wildfires via partially-interpretable neural networks
• Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán
• Evaluating forecasts for high-impact events using transformed kernel scores
• Engression: Extrapolation through the Lens of Distributional Regression
• What if? Numerical weather prediction at the crossroads
• Extrapolation-Aware Nonparametric Statistical Inference

Cited by in corpus (2)

• Can AI weather models predict out-of-distribution gray swan tropical cyclones?
• Constructing Extreme Heatwave Storylines with Differentiable Climate Models