Breaking Symmetries of the Reservoir Equations in Echo State Networks

Abstract

Reservoir computing has repeatedly been shown to be extremely successful in the prediction of nonlinear time-series. However, there is no complete understanding of the proper design of a reservoir yet. We find that the simplest popular setup has a harmful symmetry, which leads to the prediction of what we call mirror-attractor. We prove this analytically. Similar problems can arise in a general context, and we use them to explain the success or failure of some designs. The symmetry is a direct consequence of the hyperbolic tangent activation function. Further, four ways to break the symmetry are compared numerically: A bias in the output, a shift in the input, a quadratic term in the readout, and a mixture of even and odd activation functions. Firstly, we test their susceptibility to the mirror-attractor. Secondly, we evaluate their performance on the task of predicting Lorenz data with the mean shifted to zero. The short-time prediction is measured with the forecast horizon while the largest Lyapunov exponent and the correlation dimension are used to represent the climate. Finally, the same analysis is repeated on a combined dataset of the Lorenz attractor and the Halvorsen attractor, which we designed to reveal potential problems with symmetry. We find that all methods except the output bias are able to fully break the symmetry with input shift and quadratic readout performing the best overall.

arXiv comment14 pages, 10 figures, accepted by chaos

References in corpus (9)

• Recent Advances in Physical Reservoir Computing: A Review
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Attractor Reconstruction by Machine Learning
• A magnetic skyrmion as a non-linear resistive element - a potential building block for reservoir computing
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers
• Network Structure Effects in Reservoir Computers
• Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing
• Reducing network size and improving prediction stability of reservoir computing

Cited by in corpus (11)

• Learning Spatiotemporal Chaos Using Next-Generation Reservoir Computing
• Optimizing Memory in Reservoir Computers
• Time Shifts to Reduce the Size of Reservoir Computers
• Criticality in Reservoir Computer of Coupled Phase Oscillators
• Reservoir Computing with Generalized Readout based on Generalized Synchronization
• Symmetry-Aware Reservoir Computing
• Predicting high-dimensional heterogeneous time series employing generalized local states
• Tailored minimal reservoir computing: on the bidirectional connection between nonlinearities in the reservoir and in data
• Predicting three-dimensional chaotic systems with four qubit quantum systems
• Improving the prediction of spatio-temporal chaos by combining parallel reservoir computing with dimensionality reduction
• Prediction performance of random reservoirs with different topology for nonlinear dynamical systems with different number of degrees of freedom