A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit

Abstract

Despite the practical success of deep neural networks, a comprehensive theoretical framework that can predict practically relevant scores, such as the test accuracy, from knowledge of the training data is currently lacking. Huge simplifications arise in the infinite-width limit, where the number of units $N_{\ell }$ in each hidden layer ($\ell =1,\dots ,L$, being $L$ the depth of the network) far exceeds the number $P$ of training examples. This idealisation, however, blatantly departs from the reality of deep learning practice. Here, we use the toolset of statistical mechanics to overcome these limitations and derive an approximate partition function for fully-connected deep neural architectures, which encodes information about the trained models. The computation holds in the ''thermodynamic limit'' where both $N_{\ell }$ and $P$ are large and their ratio ${\alpha }_{\ell }=P/N_{\ell }$ is finite. This advance allows us to obtain (i) a closed formula for the generalisation error associated to a regression task in a one-hidden layer network with finite ${\alpha }_1$; (ii) an approximate expression of the partition function for deep architectures (via an ''effective action'' that depends on a finite number of ''order parameters''); (iii) a link between deep neural networks in the proportional asymptotic limit and Student's $t$ processes.

arXiv commentSupplemental material included

References in corpus (22)

• A mathematical theory of semantic development in deep neural networks
• The Principles of Deep Learning Theory
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Statistical Mechanics of Support Vector Networks
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Neural-net-induced Gaussian process regression for function approximation and PDE solution
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Finite Versus Infinite Neural Networks: an Empirical Study
• Learning through atypical "phase transitions" in overparameterized neural networks
• Upgrading from Gaussian Processes to Student's-T Processes
• Asymptotics of representation learning in finite Bayesian neural networks
• Statistical learning theory of structured data
• Contrasting random and learned features in deep Bayesian linear regression
• Exploring the Function Space of Deep-Learning Machines
• Counting the learnable functions of structured data
• Beyond the storage capacity: data driven satisfiability transition
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Linear Classification of Neural Manifolds with Correlated Variability
• Space of Functions Computed by Deep-Layered Machines
• Exact results on high-dimensional linear regression via statistical physics
• Critical properties of the SAT/UNSAT transitions in the classification problem of structured data

Cited by in corpus (10)

• Finite-time Lyapunov exponents of deep neural networks
• Deep neural networks have an inbuilt Occam's razor
• Random features and polynomial rules
• Coding schemes in neural networks learning classification tasks
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• Bayesian RG Flow in Neural Network Field Theories
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation