Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems

Abstract

Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is very large, e.g., for weather prediction. With small ensembles, the estimated background error covariance matrix in the EnKF algorithm suffers from sampling error, leading to an erroneous estimate of the analysis state (initial condition for the next forecast cycle). In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic flow system as a test case. This framework utilizes a pre-trained deep learning-based data-driven surrogate that inexpensively generates and evolves a large data-driven ensemble of the states of the system to accurately compute the background error covariance matrix with less sampling error. The H-EnKF framework estimates a better initial condition without the need for any ad-hoc localization strategies. H-EnKF can be extended to any ensemble-based DA algorithm, e.g., particle filters, which are currently difficult to use for high dimensional systems.

References in corpus (10)

• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• Particle Filters and Data Assimilation
• Long short-term memory embedded nudging schemes for nonlinear data assimilation of geophysical flows
• Learning physics-constrained subgrid-scale closures in the small-data regime for stable and accurate LES
• Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation
• Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers
• Exploring the Lyapunov instability properties of high-dimensional atmospheric and climate models
• Machine Learning Techniques to Construct Patched Analog Ensembles for Data Assimilation
• An Efficient and Statistically Accurate Lagrangian Data Assimilation Algorithm with Applications to Discrete Element Sea Ice Models
• BAMCAFE: A Bayesian Machine Learning Advanced Forecast Ensemble Method for Complex Turbulent Systems with Partial Observations

Cited by in corpus (8)

• A multi-model ensemble Kalman filter for data assimilation and forecasting
• CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilation
• Interpretable structural model error discovery from sparse assimilation increments using spectral bias-reduced neural networks: A quasi-geostrophic turbulence test case
• Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network
• Physically Consistent Global Atmospheric Data Assimilation with Machine Learning in Latent Space
• Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1
• State-observation augmented diffusion model for nonlinear assimilation with unknown dynamics
• Ensemble Transport Filter via Optimized Maximum Mean Discrepancy