Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model

Abstract

A novel method, based on the combination of data assimilation and machine learning is introduced. The new hybrid approach is designed for a two-fold scope: (i) emulating hidden, possibly chaotic, dynamics and (ii) predicting their future states. The method consists in applying iteratively a data assimilation step, here an ensemble Kalman filter, and a neural network. Data assimilation is used to optimally combine a surrogate model with sparse noisy data. The output analysis is spatially complete and is used as a training set by the neural network to update the surrogate model. The two steps are then repeated iteratively. Numerical experiments have been carried out using the chaotic 40-variables Lorenz 96 model, proving both convergence and statistical skill of the proposed hybrid approach. The surrogate model shows short-term forecast skill up to two Lyapunov times, the retrieval of positive Lyapunov exponents as well as the more energetic frequencies of the power density spectrum. The sensitivity of the method to critical setup parameters is also presented: the forecast skill decreases smoothly with increased observational noise but drops abruptly if less than half of the model domain is observed. The successful synergy between data assimilation and machine learning, proven here with a low-dimensional system, encourages further investigation of such hybrids with more sophisticated dynamics.

for associated code, see https://github.com/brajard/GMD-code

References in corpus (1)

• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data

Cited by in corpus (34)

• Will Artificial Intelligence supersede Earth System and Climate Models?
• Bridging observation, theory and numerical simulation of the ocean using Machine Learning
• Using machine learning to correct model error in data assimilation and forecast applications
• On closures for reduced order models $-$ A spectrum of first-principle to machine-learned avenues
• Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation
• Latent Space Data Assimilation by using Deep Learning
• Model calibration using ESEm v1.0.0 -- an open, scalable Earth System Emulator
• Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation
• Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
• Inferring unknown unknowns: Regularized bias-aware ensemble Kalman filter
• Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components
• Online learning of both state and dynamics using ensemble Kalman filters
• State, global and local parameter estimation using local ensemble Kalman filters: applications to online machine learning of chaotic dynamics
• Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations
• Equation-free surrogate modeling of geophysical flows at the intersection of machine learning and data assimilation
• 3D-Var Data Assimilation using a Variational Autoencoder
• Model selection of chaotic systems from data with hidden variables using sparse data assimilation
• Machine learning for online sea ice bias correction within global ice-ocean simulations
• FuXi Weather: A data-to-forecast machine learning system for global weather
• Deep learning delay coordinate dynamics for chaotic attractors from partial observable data
• Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case
• Reconstruction, forecasting, and stability of chaotic dynamics from partial data
• Supervised machine learning to estimate instabilities in chaotic systems: estimation of local Lyapunov exponents
• BAMCAFE: A Bayesian Machine Learning Advanced Forecast Ensemble Method for Complex Turbulent Systems with Partial Observations
• Joint calibration and mapping of satellite altimetry data using trainable variational models
• Mechanics and Thermodynamics of a New Minimal Model of the Atmosphere
• CGNSDE: Conditional Gaussian Neural Stochastic Differential Equation for Modeling Complex Systems and Data Assimilation
• Development of an offline and online hybrid model for the Integrated Forecasting System
• Heterogeneity of the Attractor of the Lorenz '96 Model: Lyapunov Analysis, Unstable Periodic Orbits, and Shadowing Properties
• A real-time digital twin of azimuthal thermoacoustic instabilities
• Analysis of a bistable climate toy model with physics-based machine learning methods
• Physics-Informed Long Short-Term Memory for Forecasting and Reconstruction of Chaos
• Hybrid machine learning data assimilation for marine biogeochemistry