The Echo Index and multistability in input-driven recurrent neural networks

Abstract

A recurrent neural network (RNN) possesses the echo state property (ESP) if, for a given input sequence, it ``forgets'' any internal states of the driven (nonautonomous) system and asymptotically follows a unique, possibly complex trajectory. The lack of ESP is conventionally understood as a lack of reliable behaviour in RNNs. Here, we show that RNNs can reliably perform computations under a more general principle that accounts only for their local behaviour in phase space. To this end, we formulate a generalisation of the ESP and introduce an echo index to characterise the number of simultaneously stable responses of a driven RNN. We show that it is possible for the echo index to change with inputs, highlighting a potential source of computational errors in RNNs due to characteristics of the inputs driving the dynamics.

arXiv commentRevised version, 43 pages, 6 figures

References in corpus (14)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An overview of gradient descent optimization algorithms
• On the difficulty of training Recurrent Neural Networks
• Synchronization in complex networks
• Attractor Reconstruction by Machine Learning
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Full-Capacity Unitary Recurrent Neural Networks
• Using reservoir computers to distinguish chaotic signals
• Local Dynamics in Trained Recurrent Neural Networks
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Interpreting recurrent neural networks behaviour via excitable network attractors
• Detection of Generalized Synchronization using Echo State Networks
• Nonautonomous driving induces stability in network of identical oscillators
• Chasing the Echo State Property

Cited by in corpus (6)

• Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem
• Learn to Synchronize, Synchronize to Learn
• Embedding Information onto a Dynamical System
• Learning unidirectional coupling using echo-state network
• Echoes of the Past: A Unified Perspective on Fading memory and Echo States
• (Thesis) Reservoir Computing With Dynamical Systems