Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM

Abstract

In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/large-scale ($X$), intermediate ($Y$), and fast/small-scale ($Z$) processes. For training or testing, only $X$ is available; $Y$ and $Z$ are never known or used. We show that RC-ESN substantially outperforms ANN and RNN-LSTM for short-term prediction, e.g., accurately forecasting the chaotic trajectories for hundreds of numerical solver's time steps, equivalent to several Lyapunov timescales. The RNN-LSTM and ANN show some prediction skills as well; RNN-LSTM bests ANN. Furthermore, even after losing the trajectory, data predicted by RC-ESN and RNN-LSTM have probability density functions (PDFs) that closely match the true PDF, even at the tails. The PDF of the data predicted using ANN, however, deviates from the true PDF. Implications, caveats, and applications to data-driven and data-assisted surrogate modeling of complex nonlinear dynamical systems such as weather/climate are discussed.

Some changes, in Figures, addition of an appendix etc has been done

References in corpus (22)

• A Data-Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition
• On Dynamic Mode Decomposition: Theory and Applications
• Turbulence Modeling in the Age of Data
• Deep learning to represent sub-grid processes in climate models
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Ergodic theory, Dynamic Mode Decomposition and Computation of Spectral Properties of the Koopman operator
• Attractor Reconstruction by Machine Learning
• Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks
• Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations
• Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model
• Using machine learning to parameterize moist convection: potential for modeling of climate, climate change and extreme events
• Analog forecasting of extreme-causing weather patterns using deep learning
• Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model
• A test case for application of convolutional neural networks to spatio-temporal climate data: Re-identifying clustered weather patterns
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• Enforcing Statistical Constraints in Generative Adversarial Networks for Modeling Chaotic Dynamical Systems
• Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correction
• Data-driven super-parameterization using deep learning: Experimentation with multi-scale Lorenz 96 systems and transfer-learning
• Deep learning of mixing by two 'atoms' of stratified turbulence
• Data-driven reduced modelling of turbulent Rayleigh-Benard convection using DMD-enhanced Fluctuation-Dissipation Theorem
• Reduced-order modeling of fully turbulent buoyancy-driven flows using the Green's function method

Cited by in corpus (41)

• Transformers for Modeling Physical Systems
• Stable a posteriori LES of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher Re via transfer learning
• Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems
• Data-driven subgrid-scale modeling of forced Burgers turbulence using deep learning with generalization to higher Reynolds numbers via transfer learning
• Integrating Recurrent Neural Networks with Data Assimilation for Scalable Data-Driven State Estimation
• Reservoir Computing as a Tool for Climate Predictability Studies
• Data-driven super-parameterization using deep learning: Experimentation with multi-scale Lorenz 96 systems and transfer-learning
• Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computing
• Using Machine Learning to Anticipate Tipping Points and Extrapolate to Post-Tipping Dynamics of Non-Stationary Dynamical Systems
• Breaking Symmetries of the Reservoir Equations in Echo State Networks
• Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers
• Climbing down Charney's ladder: Machine Learning and the post-Dennard era of computational climate science
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components
• Accuracy of neural networks for the simulation of chaotic dynamics: precision of training data vs precision of the algorithm
• Learn to Synchronize, Synchronize to Learn
• Predicting Critical Transitions in Multiscale Dynamical Systems Using Reservoir Computing
• Overcoming the Coherence Time Barrier in Quantum Machine Learning on Temporal Data
• Forecasting trends in food security with real time data
• Predicting rare events using neural networks and short-trajectory data
• Discovery of interpretable structural model errors by combining Bayesian sparse regression and data assimilation: A chaotic Kuramoto-Sivashinsky test case
• Nonequilibrium Statistical Mechanics and Optimal Prediction of Partially-Observed Complex Systems
• Symmetry-Aware Reservoir Computing
• A Causality-Based Learning Approach for Discovering the Underlying Dynamics of Complex Systems from Partial Observations with Stochastic Parameterization
• BAMCAFE: A Bayesian Machine Learning Advanced Forecast Ensemble Method for Complex Turbulent Systems with Partial Observations
• Neural Echo State Network using oscillations of gas bubbles in water
• Model-assisted deep learning of rare extreme events from partial observations
• CEBoosting: Online Sparse Identification of Dynamical Systems with Regime Switching by Causation Entropy Boosting
• Mechanics and Thermodynamics of a New Minimal Model of the Atmosphere
• Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features
• Heterogeneity of the Attractor of the Lorenz '96 Model: Lyapunov Analysis, Unstable Periodic Orbits, and Shadowing Properties
• Analysis of a bistable climate toy model with physics-based machine learning methods
• Recurrent Neural Networks for Partially Observed Dynamical Systems
• On the Universal Transformation of Data-Driven Models to Control Systems
• Improving Long-term Autoregressive Spatiotemporal Predictions: A Proof of Concept with Fluid Dynamics
• Data-Driven Forecasting of Non-Equilibrium Solid-State Dynamics
• Model-free Forecasting of Rogue Waves using Reservoir Computing
• Machine-learning Closure for Vlasov-Poisson Dynamics in Fourier-Hermite Space
• Circuit design in biology and machine learning. I. Random networks and dimensional reduction
• A comparison of data-driven approaches to build low-dimensional ocean models
• Direct data-driven forecast of local turbulent heat flux in Rayleigh-Bénard convection
• Phase space integrity in neural network models of Hamiltonian dynamics: A Lagrangian descriptor approach