Reservoir Computing meets Recurrent Kernels and Structured Transforms

Abstract

Reservoir Computing is a class of simple yet efficient Recurrent Neural Networks where internal weights are fixed at random and only a linear output layer is trained. In the large size limit, such random neural networks have a deep connection with kernel methods. Our contributions are threefold: a) We rigorously establish the recurrent kernel limit of Reservoir Computing and prove its convergence. b) We test our models on chaotic time series prediction, a classic but challenging benchmark in Reservoir Computing, and show how the Recurrent Kernel is competitive and computationally efficient when the number of data points remains moderate. c) When the number of samples is too large, we leverage the success of structured Random Features for kernel approximation by introducing Structured Reservoir Computing. The two proposed methods, Recurrent Kernel and Structured Reservoir Computing, turn out to be much faster and more memory-efficient than conventional Reservoir Computing.

References in corpus (19)

• On the difficulty of training Recurrent Neural Networks
• Recent Advances in Physical Reservoir Computing: A Review
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Recent Advances in Recurrent Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Unitary Evolution Recurrent Neural Networks
• Generalization Properties of Learning with Random Features
• Optical Reservoir Computing using multiple light scattering for chaotic systems prediction
• Random Feature Maps for Dot Product Kernels
• Random Projections through multiple optical scattering: Approximating kernels at the speed of light
• Using a reservoir computer to learn chaotic attractors, with applications to chaos synchronisation and cryptography
• Orthogonal Random Features
• The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
• Learning with SGD and Random Features
• Kernel computations from large-scale random features obtained by Optical Processing Units
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• ACDC: A Structured Efficient Linear Layer
• Recurrent Kernel Networks
• Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods

Cited by in corpus (1)

• SpaRCe: Improved Learning of Reservoir Computing Systems through Sparse Representations