Local Dynamics in Trained Recurrent Neural Networks

Abstract

Learning a task induces connectivity changes in neural circuits, thereby changing their dynamics. To elucidate task related neural dynamics we study trained Recurrent Neural Networks. We develop a Mean Field Theory for Reservoir Computing networks trained to have multiple fixed point attractors. Our main result is that the dynamics of the network's output in the vicinity of attractors is governed by a low order linear Ordinary Differential Equation. Stability of the resulting ODE can be assessed, predicting training success or failure. As a consequence, networks of Rectified Linear (RLU) and of sigmoidal nonlinearities are shown to have diametrically different properties when it comes to learning attractors. Furthermore, a characteristic time constant, which remains finite at the edge of chaos, offers an explanation of the network's output robustness in the presence of variability of the internal neural dynamics. Finally, the proposed theory predicts state dependent frequency selectivity in network response.

References in corpus (3)

• Transition to chaos in random neuronal networks
• Dynamics of Random Neural Networks with Bistable Units
• Intrinsically-generated fluctuating activity in excitatory-inhibitory networks

Cited by in corpus (23)

• Linking connectivity, dynamics and computations in low-rank recurrent neural networks
• Supervised Learning in Spiking Neural Networks with FORCE Training
• Dynamics of random recurrent networks with correlated low-rank structure
• Interpreting recurrent neural networks behaviour via excitable network attractors
• Dimension of activity in random neural networks
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Quality of internal representation shapes learning performance in feedback neural networks
• Learn to Synchronize, Synchronize to Learn
• Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
• The Echo Index and multistability in input-driven recurrent neural networks
• Macroscopic Fluctuations Emerge in Balanced Networks with Incomplete Recurrent Alignment
• Predictive coding in balanced neural networks with noise, chaos and delays
• Effective models and predictability of chaotic multiscale systems via machine learning
• Frustration induced transient chaos, fractal and riddled basins in coupled limit cycle oscillators
• Machine Learning assisted Chimera and Solitary states in Networks
• A characterization of the Edge of Criticality in Binary Echo State Networks
• Short term memory by transient oscillatory dynamics in recurrent neural networks
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks
• Thalamocortical motor circuit insights for more robust hierarchical control of complex sequences
• Evolutionary aspects of Reservoir Computing
• Learning recurrent dynamics in spiking networks
• A geometrical analysis of global stability in trained feedback networks