Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Abstract

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory--nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behaviour in phase space characterised by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking.

References in corpus (12)

• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks
• Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model
• Local Dynamics in Trained Recurrent Neural Networks
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Interpreting recurrent neural networks behaviour via excitable network attractors
• The difference between memory and prediction in linear recurrent networks
• Comparison between DeepESNs and gated RNNs on multivariate time-series prediction
• Memory and Information Processing in Recurrent Neural Networks
• A characterization of the Edge of Criticality in Binary Echo State Networks
• Reservoir computing approaches for representation and classification of multivariate time series
• Reservoir Computing on the Hypersphere

Cited by in corpus (8)

• Physics-Informed Echo State Networks
• Memory and forecasting capacities of nonlinear recurrent networks
• Reducing network size and improving prediction stability of reservoir computing
• Risk bounds for reservoir computing
• Reservoir computing based on solitary-like waves dynamics of film flows: a proof of concept
• Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems
• Scope and Sense of Explainability for AI-Systems
• Learnability Window in Gated Recurrent Neural Networks