Embedding and Approximation Theorems for Echo State Networks

Abstract

Echo State Networks (ESNs) are a class of single layer recurrent neural networks that have enjoyed recent attention. In this paper we prove that a suitable ESN, trained on a series of measurements of an invertible dynamical system, induces a C1 map from the dynamical system's phase space to the ESN's reservoir space. We call this the Echo State Map. We then prove that the Echo State Map is generically an embedding with positive probability. Under additional mild assumptions, we further conjecture that the Echo State Map is almost surely an embedding. For sufficiently large, and specially structured, but still randomly generated ESNs, we prove that there exists a linear readout layer that allows the ESN to predict the next observation of a dynamical system arbitrarily well. Consequently, if the dynamical system under observation is structurally stable then the trained ESN will exhibit dynamics that are topologically conjugate to the future behaviour of the observed dynamical system. Our theoretical results connect the theory of ESNs to the delay-embedding literature for dynamical systems, and are supported by numerical evidence from simulations of the traditional Lorenz equations. The simulations confirm that, from a one dimensional observation function, an ESN can accurately infer a range of geometric and topological features of the dynamics such as the eigenvalues of equilibrium points, Lyapunov exponents and homology groups.

24 pages, 9 figures

References in corpus (8)

• Recent Advances in Physical Reservoir Computing: A Review
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Training Echo State Networks with Regularization through Dimensionality Reduction
• Exploring the Topology of Dynamical Reconstructions
• Data-driven Reconstruction of Nonlinear Dynamics from Sparse Observation
• Bayesian optimization of hyper-parameters in reservoir computing
• Approximation Bounds for Random Neural Networks and Reservoir Systems
• TikZ-network manual

Cited by in corpus (28)

• On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD
• Do Reservoir Computers Work Best at the Edge of Chaos?
• Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems
• Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations
• Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem
• Optimizing Memory in Reservoir Computers
• Learning strange attractors with reservoir systems
• Predicting Critical Transitions in Multiscale Dynamical Systems Using Reservoir Computing
• Learn to Synchronize, Synchronize to Learn
• Time Shifts to Reduce the Size of Reservoir Computers
• Chaotic attractor reconstruction using small reservoirs -- the influence of topology
• Stability and Memory-loss go Hand-in-Hand: Three Results in Dynamics & Computation
• Stochastic Reservoir Computers
• Oscillations enhance time-series prediction in reservoir computing with feedback
• Limits of Learning Dynamical Systems
• Universality and approximation bounds for echo state networks with random weights
• Thermodynamic limit in learning period three
• Toward the Fully Physics-Informed Echo State Network -- an ODE Approximator Based on Recurrent Artificial Neurons
• Attractor-merging Crises and Intermittency in Reservoir Computing
• Using Echo State Networks to Approximate Value Functions for Control
• Echoes of the Past: A Unified Perspective on Fading memory and Echo States
• Can neural networks predict dynamics they have never seen?
• Investigating the Surrogate Modeling Capabilities of Continuous Time Echo State Networks
• Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
• (Thesis) Reservoir Computing With Dynamical Systems
• Dynamics of reservoir computing for crises prediction
• Phase space integrity in neural network models of Hamiltonian dynamics: A Lagrangian descriptor approach
• Finite-time Lyaponov analysis of a trained reservoir computer