Dimension of activity in random neural networks

Abstract

Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. Understanding how biological and machine-learning networks function and learn requires knowledge of the structure of this coordinated activity, information contained, for example, in cross covariances between units. Self-consistent dynamical mean field theory (DMFT) has elucidated several features of random neural networks -- in particular, that they can generate chaotic activity -- however, a calculation of cross covariances using this approach has not been provided. Here, we calculate cross covariances self-consistently via a two-site cavity DMFT. We use this theory to probe spatiotemporal features of activity coordination in a classic random-network model with independent and identically distributed (i.i.d.) couplings, showing an extensive but fractionally low effective dimension of activity and a long population-level timescale. Our formulae apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings. As an example of the latter, we analyze the case of partially symmetric couplings.

8 pages, 6 figures; clarified derivation

References in corpus (8)

• The Principles of Deep Learning Theory
• Transition to chaos in random networks with cell-type-specific connectivity
• Dynamics of Random Neural Networks with Bistable Units
• Quality of internal representation shapes learning performance in feedback neural networks
• Unified field theoretical approach to deep and recurrent neuronal networks
• Dynamical mean-field theory: from ecosystems to reaction networks
• Extended Anderson Criticality in Heavy-Tailed Neural Networks
• Soft-margin classification of object manifolds

Cited by in corpus (6)

• An optimization-based equilibrium measure describes non-equilibrium steady state dynamics: application to edge of chaos
• Dynamical Theory for Adaptive Systems
• Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks
• Discontinuous transition to chaos in a canonical random neural network
• Hierarchy of chaotic dynamics in random modular networks
• Statistics of correlations in nonlinear recurrent neural networks