Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems

Abstract

Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure $\mu$) using Tikhonov least squares regression against a set of targets, will approximate the target function in the $L^2(\mu )$ norm. In the special case that the targets are future observations, the ESN is learning the next step map, which allows time series forecasting. We demonstrate the theory numerically by training an ESN using Tikhonov least squares on a sequence of scalar observations of the Lorenz system.

arXiv comment11 pages, 5 figures

References in corpus (7)

• Recent Advances in Physical Reservoir Computing: A Review
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Embedding and Approximation Theorems for Echo State Networks
• The Lorenz attractor is mixing
• Training Echo State Networks with Regularization through Dimensionality Reduction
• Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem
• Data-driven Reconstruction of Nonlinear Dynamics from Sparse Observation

Cited by in corpus (15)

• Next Generation Reservoir Computing
• On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD
• A Survey on Reservoir Computing and its Interdisciplinary Applications Beyond Traditional Machine Learning
• Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem
• Learning strange attractors with reservoir systems
• Catch-22s of reservoir computing
• Gradient-free optimization of chaotic acoustics with reservoir computing
• Reservoir Computing for Macroeconomic Forecasting with Mixed Frequency Data
• Quantum Next Generation Reservoir Computing: An Efficient Quantum Algorithm for Forecasting Quantum Dynamics
• Limits of Learning Dynamical Systems
• Toward the Fully Physics-Informed Echo State Network -- an ODE Approximator Based on Recurrent Artificial Neurons
• Using Echo State Networks to Approximate Value Functions for Control
• Analysis on reservoir activation with the nonlinearity harnessed from solution-processed molybdenum disulfide
• Phase space integrity in neural network models of Hamiltonian dynamics: A Lagrangian descriptor approach
• (Thesis) Reservoir Computing With Dynamical Systems