Optimal short-term memory before the edge of chaos in driven random recurrent networks

Abstract

The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated by mean-field theory. The combination of a small input strength and mean-field assumptions makes it possible to derive an approximate expression for the conditional probability density of the state of a neuron given a past input signal. From this conditional probability density, we can analytically calculate short-term memory measures, such as memory capacity, mutual information, and Fisher information, and determine the relationships among these measures, which have not been clarified to date to the best of our knowledge. We show that the network contribution of these short-term memory measures peaks before the edge of chaos, where the dynamics of input-driven networks is stable but corresponding systems without input signals are unstable.

References in corpus (6)

• Harnessing disordered quantum dynamics for machine learning
• Short-Term Memory in Orthogonal Neural Networks
• Optimal sequence memory in driven random networks
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Mean Field Theory of Dynamical Systems Driven by External Signals
• The difference between memory and prediction in linear recurrent networks

Cited by in corpus (7)

• Physical reservoir computing -- An introductory perspective
• Learning Temporal Quantum Tomography
• Suppression of chaos in a partially driven recurrent neural network
• Edge of chaos as a guiding principle for modern neural network training
• Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks
• Attractor-merging Crises and Intermittency in Reservoir Computing
• Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks