3D-Var Data Assimilation using a Variational Autoencoder

Abstract

Data assimilation of atmospheric observations traditionally relies on variational and Kalman filter methods. Here, an alternative neural-network data assimilation (NNDA) with variational autoencoder (VAE) is proposed. The three-dimensional variational (3D-Var) data assimilation cost function is utilised to determine the analysis that optimally fuses simulated observations and the encoded short-range persistence forecast (background), accounting for their errors. The minimisation is performed in the reduced-order latent space, discovered by the VAE. The variational problem is auto-differentiable, simplifying the computation of the cost function gradient necessary for efficient minimisation. We demonstrate that the background-error covariance ($\mathbf{B}$) matrix measured and represented in the latent space is quasi-diagonal. The background-error covariances in the grid-point space are flow-dependent, evolving seasonally and depending on the current state of the atmosphere. Data assimilation experiments with a single temperature observation in the lower troposphere indicate that the $\mathbf{B}$-matrix simultaneously describes both tropical and extratropical background-error covariances.

arXiv commentPeer-reviewed version of this paper has been published in QJRMS (https://doi.org/10.1002/qj.4708). The uploaded version in this repository is not the same as the peer-reviewed version

References in corpus (14)

• An Introduction to Variational Autoencoders
• Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions
• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model
• Towards a Robust Parameterization for Conditioning Facies Models Using Deep Variational Autoencoders and Ensemble Smoother
• Using machine learning to correct model error in data assimilation and forecast applications
• Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization
• Combining distribution-based neural networks to predict weather forecast probabilities
• A reduced-order strategy for 4D-Var data assimilation
• Attention-based Convolutional Autoencoders for 3D-Variational Data Assimilation
• Machine Learning Emulation of 3D Cloud Radiative Effects
• Analog ensemble data assimilation and a method for constructing analogs with variational autoencoders
• Combining data assimilation and machine learning to estimate parameters of a convective-scale model
• Variational Diffusion Autoencoders with Random Walk Sampling

Cited by in corpus (2)

• Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations
• Physically Consistent Global Atmospheric Data Assimilation with Machine Learning in Latent Space