On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD

Abstract

Machine learning has become a widely popular and successful paradigm, including in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical. Artificial neural networks (ANN) have evolved as a clear leader amongst many machine learning approaches, and recurrent neural networks (RNN) are considered to be especially well suited for forecasting dynamical systems. In this setting, the echo state networks (ESN) or reservoir computer (RC) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only read-out weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless an RC succeeds to make high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all, despite randomly selected weights. We explicitly connect the RC with linear activation and linear read-out to well developed time-series literature on vector autoregressive averages (VAR) that includes theorems on representability through the WOLD theorem, which already perform reasonably for short term forecasts. In the case of a linear activation and now popular quadratic read-out RC, we explicitly connect to a nonlinear VAR (NVAR), which performs quite well. Further, we associate this paradigm to the now widely popular dynamic mode decomposition (DMD), and thus these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.

arXiv comment34 pages, 9 figures

References in corpus (17)

• On the difficulty of training Recurrent Neural Networks
• A Data-Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition
• Applied Koopmanism
• Ergodic theory, Dynamic Mode Decomposition and Computation of Spectral Properties of the Koopman operator
• Extended dynamic mode decomposition with dictionary learning: a data-driven adaptive spectral decomposition of the Koopman operator
• Attractor Reconstruction by Machine Learning
• Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks
• Causal Network Inference by Optimal Causation Entropy
• Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers
• Embedding and Approximation Theorems for Echo State Networks
• Deep learning algorithm for data-driven simulation of noisy dynamical system
• Using a reservoir computer to learn chaotic attractors, with applications to chaos synchronisation and cryptography
• Network Structure Effects in Reservoir Computers
• Rapid Time Series Prediction with a Hardware-Based Reservoir Computer
• Chaos in time delay systems, an educational review
• Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems
• Model-free prediction of noisy chaotic time series by deep learning

Cited by in corpus (32)

• Next Generation Reservoir Computing
• Model-free tracking control of complex dynamical trajectories with machine learning
• A Survey on Reservoir Computing and its Interdisciplinary Applications Beyond Traditional Machine Learning
• Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
• Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computing
• Reservoir Computing Approach to Quantum State Measurement
• Learning unseen coexisting attractors
• Controlling Chaos Using Edge Computing Hardware
• Catch-22s of reservoir computing
• Model-free prediction of emergence of extreme events in a parametrically driven nonlinear dynamical system by Deep Learning
• Machine Learning Link Inference of Noisy Delay-coupled Networks with Opto-Electronic Experimental Tests
• Hybridizing Traditional and Next-Generation Reservoir Computing to Accurately and Efficiently Forecast Dynamical Systems
• Criticality in Reservoir Computer of Coupled Phase Oscillators
• Symbolic Regression via Neural Networks
• Nonequilibrium Statistical Mechanics and Optimal Prediction of Partially-Observed Complex Systems
• Reservoir Computing with Generalized Readout based on Generalized Synchronization
• Two methods to approximate the Koopman operator with a reservoir computer
• Reservoir Computers with Random and Optimized Time-Shifts
• Quantum Next Generation Reservoir Computing: An Efficient Quantum Algorithm for Forecasting Quantum Dynamics
• Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features
• Quantum Next-Generation Reservoir Computing and Its Quantum Optical Implementation
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• Learning Beyond Experience: Generalizing to Unseen State Space with Reservoir Computing
• Data-Driven Forecasting of Non-Equilibrium Solid-State Dynamics
• Reservoir Computers Modal Decomposition and Optimization
• Improving the prediction of spatio-temporal chaos by combining parallel reservoir computing with dimensionality reduction
• Data-driven stability analysis in a multi-element supercritical Liquid Oxygen-methane combustor
• Dynamics of reservoir computing for crises prediction
• Finite-time Lyaponov analysis of a trained reservoir computer
• (Thesis) Reservoir Computing With Dynamical Systems
• Nonlinear Autoregression with Convergent Dynamics on Novel Computational Platforms
• Physical reservoir computing using finitely-sampled quantum systems