Neural networks for post-processing ensemble weather forecasts

Abstract

Ensemble weather predictions require statistical post-processing of systematic errors to obtain reliable and accurate probabilistic forecasts. Traditionally, this is accomplished with distributional regression models in which the parameters of a predictive distribution are estimated from a training period. We propose a flexible alternative based on neural networks that can incorporate nonlinear relationships between arbitrary predictor variables and forecast distribution parameters that are automatically learned in a data-driven way rather than requiring pre-specified link functions. In a case study of 2-meter temperature forecasts at surface stations in Germany, the neural network approach significantly outperforms benchmark post-processing methods while being computationally more affordable. Key components to this improvement are the use of auxiliary predictor variables and station-specific information with the help of embeddings. Furthermore, the trained neural network can be used to gain insight into the importance of meteorological variables thereby challenging the notion of neural networks as uninterpretable black boxes. Our approach can easily be extended to other statistical post-processing and forecasting problems. We anticipate that recent advances in deep learning combined with the ever-increasing amounts of model and observation data will transform the post-processing of numerical weather forecasts in the coming decade.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• TensorFlow: A system for large-scale machine learning
• Deep learning to represent sub-grid processes in climate models
• Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets
• Entity Embeddings of Categorical Variables
• Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling
• Probabilistic Quantitative Precipitation Forecasting Using Ensemble Model Output Statistics
• Photometric redshift estimation via deep learning
• Log-normal distribution based EMOS models for probabilistic wind speed forecasting
• Comparison of nonhomogeneous regression models for probabilistic wind speed forecasting
• Probabilistic wind speed forecasting on a grid based on ensemble model output statistics
• Combining predictive distributions for statistical post-processing of ensemble forecasts
• Distributional Regression Forests for Probabilistic Precipitation Forecasting in Complex Terrain
• Mixture EMOS model for calibrating ensemble forecasts of wind speed
• Forest-based methods and ensemble model output statistics for rainfall ensemble forecasting
• Similarity-based semi-local estimation of EMOS models

Cited by in corpus (60)

• A review of machine learning applications in wildfire science and management
• WeatherBench: A benchmark dataset for data-driven weather forecasting
• Neural General Circulation Models for Weather and Climate
• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• A test case for application of convolutional neural networks to spatio-temporal climate data: Re-identifying clustered weather patterns
• Deep Learning for Post-Processing Ensemble Weather Forecasts
• Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for WeatherBench
• Bridging observation, theory and numerical simulation of the ocean using Machine Learning
• A review of predictive uncertainty estimation with machine learning
• Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correction
• Short-term precipitation prediction using deep learning
• Distributional Regression Forests for Probabilistic Precipitation Forecasting in Complex Terrain
• Post-processing numerical weather prediction ensembles for probabilistic solar irradiance forecasting
• Statistical post-processing of wind speed forecasts using convolutional neural networks
• Combining distribution-based neural networks to predict weather forecast probabilities
• Forest-based methods and ensemble model output statistics for rainfall ensemble forecasting
• Isotonic Distributional Regression
• A framework for probabilistic weather forecast post-processing across models and lead times using machine learning
• Copula-based synthetic data augmentation for machine-learning emulators
• Machine learning for total cloud cover prediction
• Deep Learning for Spatio-Temporal Data Mining: A Survey
• Coupled online learning as a way to tackle instabilities and biases in neural network parameterizations
• Comparison of multivariate post-processing methods using global ECMWF ensemble forecasts
• Machine Learning Emulation of Urban Land Surface Processes
• A neural network-based scale-adaptive cloud-fraction scheme for GCMs
• Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications
• Generative machine learning methods for multivariate ensemble post-processing
• Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison
• Countdown Regression: Sharp and Calibrated Survival Predictions
• Deep learning for improved global precipitation in numerical weather prediction systems
• Probabilistic Temperature Forecasting with a Heteroscedastic Autoregressive Ensemble Postprocessing model
• Evaluation of Machine Learning Techniques for Forecast Uncertainty Quantification
• Postprocessing of Ensemble Weather Forecasts Using Permutation-invariant Neural Networks
• Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression
• Truncated generalized extreme value distribution based EMOS model for calibration of wind speed ensemble forecasts
• Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation
• A two-step machine learning approach to statistical post-processing of weather forecasts for power generation
• Machine learning-based probabilistic forecasting of solar irradiance in Chile
• Calibration of wind speed ensemble forecasts for power generation
• Predicting Weather Uncertainty with Deep Convnets
• Statistical post-processing of visibility ensemble forecasts
• Distributional regression and its evaluation with the CRPS: Bounds and convergence of the minimax risk
• Parametric post-processing of dual-resolution precipitation forecasts
• Multivariate Probabilistic Regression with Natural Gradient Boosting
• Climate Modeling with Neural Diffusion Equations
• Clustering-based spatial interpolation of parametric post-processing models
• Skillful Twelve Hour Precipitation Forecasts using Large Context Neural Networks
• Improving probabilistic forecasts of extreme wind speeds by training statistical post-processing models with weighted scoring rules
• Probabilistic prediction of the time to hard freeze using seasonal weather forecasts and survival time methods
• Post-processing Multi-Model Medium-Term Precipitation Forecasts Using Convolutional Neural Networks
• Parametric model for post-processing visibility ensemble forecasts
• Post-processing of ensemble photovoltaic power forecasts with distributional and quantile regression methods
• A Composite-Loss Graph Neural Network for the Multivariate Post-Processing of Ensemble Weather Forecasts
• Scoring rule nets: beyond mean target prediction in multivariate regression
• Enhancing multivariate post-processed visibility predictions utilizing CAMS forecasts
• Self-Attentive Ensemble Transformer: Representing Ensemble Interactions in Neural Networks for Earth System Models
• High-resolution Probabilistic Precipitation Prediction for use in Climate Simulations
• Achieving Skilled and Reliable Daily Probabilistic Forecasts of Wind Power at Subseasonal-to-Seasonal Timescales over France
• Joint Forecasting and Interpolation of Graph Signals Using Deep Learning
• Leveraging deterministic weather forecasts for in-situ probabilistic temperature predictions via deep learning