Dimension of activity in random neural networks

Abstract

Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. Understanding how biological and machine-learning networks function and learn requires knowledge of the structure of this coordinated activity, information contained, for example, in cross covariances between units. Self-consistent dynamical mean field theory (DMFT) has elucidated several features of random neural networks -- in particular, that they can generate chaotic activity -- however, a calculation of cross covariances using this approach has not been provided. Here, we calculate cross covariances self-consistently via a two-site cavity DMFT. We use this theory to probe spatiotemporal features of activity coordination in a classic random-network model with independent and identically distributed (i.i.d.) couplings, showing an extensive but fractionally low effective dimension of activity and a long population-level timescale. Our formulae apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings. As an example of the latter, we analyze the case of partially symmetric couplings.

arXiv comment8 pages, 6 figures; clarified derivation

References in corpus (29)

• Nonequilibrium dynamical mean-field theory and its applications
• Linking connectivity, dynamics and computations in low-rank recurrent neural networks
• Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks
• The Principles of Deep Learning Theory
• Transition to chaos in random neuronal networks
• Transition to chaos in random networks with cell-type-specific connectivity
• Classification and Geometry of General Perceptual Manifolds
• Distributions of covariances as a window into the operational regime of neuronal networks
• Two types of criticality in the brain
• Numerical implementation of dynamical mean field theory for disordered systems: application to the Lotka-Volterra model of ecosystems
• Dynamics of Random Neural Networks with Bistable Units
• Optimal sequence memory in driven random networks
• Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks
• Path Integral Approach to Random Neural Networks
• Local Dynamics in Trained Recurrent Neural Networks
• Statistical mechanics of complex neural systems and high dimensional data
• A unified view on weakly correlated recurrent networks
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Out-of-equilibrium dynamical mean-field equations for the perceptron model
• How single neuron properties shape chaotic dynamics and signal transmission in random neural networks
• Contrasting the effects of adaptation and synaptic filtering on the timescales of dynamics in recurrent networks
• Quality of internal representation shapes learning performance in feedback neural networks
• Transient chaotic dimensionality expansion by recurrent networks
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Unified field theoretical approach to deep and recurrent neuronal networks
• Dynamical mean-field theory: from ecosystems to reaction networks
• Large Deviations Approach to Random Recurrent Neuronal Networks: Parameter Inference and Fluctuation-Induced Transitions
• Extended Anderson Criticality in Heavy-Tailed Neural Networks
• Soft-margin classification of object manifolds

Cited by in corpus (6)

• An optimization-based equilibrium measure describes non-equilibrium steady state dynamics: application to edge of chaos
• Dynamical Theory for Adaptive Systems
• Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks
• Discontinuous transition to chaos in a canonical random neural network
• Hierarchy of chaotic dynamics in random modular networks
• Statistics of correlations in nonlinear recurrent neural networks