BAMCAFE: A Bayesian Machine Learning Advanced Forecast Ensemble Method for Complex Turbulent Systems with Partial Observations

Abstract

Ensemble forecast based on physics-informed models is one of the most widely used forecast algorithms for complex turbulent systems. A major difficulty in such a method is the model error that is ubiquitous in practice. Data-driven machine learning (ML) forecasts can reduce the model error but they often suffer from the partial and noisy observations. In this paper, a simple but effective Bayesian machine learning advanced forecast ensemble (BAMCAFE) method is developed, which combines an available imperfect physics-informed model with data assimilation (DA) to facilitate the ML ensemble forecast. In the BAMCAFE framework, a Bayesian ensemble DA is applied to create the training data of the ML model, which reduces the intrinsic error in the imperfect physics-informed model simulations and provides the training data of the unobserved variables. Then a generalized DA is employed for the initialization of the ML ensemble forecast. In addition to forecasting the optimal point-wise value, the BAMCAFE also provides an effective approach of quantifying the forecast uncertainty utilizing a non-Gaussian probability density function that characterizes the intermittency and extreme events. It is shown using a two-layer Lorenz 96 model that the BAMCAFE method can significant improve the forecasting skill compared to the typical reduced-order imperfect models with bare truncation or stochastic parameterization for both the observed and unobserved large-scale variables. It is also shown via a nonlinear conceptual model that the BAMCAFE leads to a comparable non-Gaussian forecast uncertainty as the perfect model while the associated imperfect physics-informed model suffers from large forecast biases.

arXiv commentThe following article has been accepted by Chaos: An Interdisciplinary Journal of Nonlinear Science

References in corpus (20)

• Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks
• Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere
• Deep Learning for Physical Processes: Incorporating Prior Scientific Knowledge
• Neural Network with Unbounded Activation Functions is Universal Approximator
• Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model
• Data-driven medium-range weather prediction with a Resnet pretrained on climate simulations: A new model for WeatherBench
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• Using machine learning to correct model error in data assimilation and forecast applications
• The ECMWF Ensemble Prediction System: Looking Back (more than) 25 Years and Projecting Forward 25 Years
• Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling
• Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization
• Data-Driven Variational Multiscale Reduced Order Models
• Ensemble Kalman methods for high-dimensional hierarchical dynamic space-time models
• Component-Wise Markov Chain Monte Carlo: Uniform and Geometric Ergodicity under Mixing and Composition
• Data-driven super-parameterization using deep learning: Experimentation with multi-scale Lorenz 96 systems and transfer-learning
• Efficient Statistically Accurate Algorithms for the Fokker-Planck Equation in Large Dimensions
• Stochastic superparameterization in quasigeostrophic turbulence
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components
• Data-Driven Correction Reduced Order Models for the Quasi-Geostrophic Equations: A Numerical Investigation
• Combining Ensemble Kalman Filter and Reservoir Computing to predict spatio-temporal chaotic systems from imperfect observations and models

Cited by in corpus (6)

• Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems
• Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case
• Conditional Gaussian Nonlinear System: a Fast Preconditioner and a Cheap Surrogate Model For Complex Nonlinear Systems
• A Causality-Based Learning Approach for Discovering the Underlying Dynamics of Complex Systems from Partial Observations with Stochastic Parameterization
• Full and Reduced Order Model Consistency of the Nonlinearity Discretization in Incompressible Flows
• CGKN: A Deep Learning Framework for Modeling Complex Dynamical Systems and Efficient Data Assimilation