Unified field theoretical approach to deep and recurrent neuronal networks

Abstract

Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, $n\to \infty$. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in $1/n$ and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths $n$.

arXiv commentRevision including next-to-leading-order corrections

References in corpus (13)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Array Programming with NumPy
• Quantum Field Theory in the Large N Limit: a review
• Optimal sequence memory in driven random networks
• Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks
• Path Integral Approach to Random Neural Networks
• Path integral methods for the dynamics of stochastic and disordered systems
• Neural Networks and Quantum Field Theory
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Contrasting random and learned features in deep Bayesian linear regression
• Large Deviations Approach to Random Recurrent Neuronal Networks: Parameter Inference and Fluctuation-Induced Transitions
• A Microscopic Theory of Intrinsic Timescales in Spiking Neural Networks
• Space of Functions Computed by Deep-Layered Machines

Cited by in corpus (6)

• Dimension of activity in random neural networks
• Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings
• Incorporating Heterogeneous Interactions for Ecological Biodiversity
• Decomposing neural networks as mappings of correlation functions
• Eight challenges in developing theory of intelligence
• Coding schemes in neural networks learning classification tasks