Statistical mechanics of extensive-width Bayesian neural networks near interpolation

Abstract

For three decades statistical mechanics has been providing a framework to analyse neural networks. However, the theoretically tractable models, e.g., perceptrons, random features models and kernel machines, or multi-index models and committee machines with few neurons, remained simple compared to those used in applications. In this paper we help reducing the gap between practical networks and their theoretical understanding through a statistical physics analysis of the supervised learning of a two-layer fully connected network with generic weight distribution and activation function, whose hidden layer is large but remains proportional to the inputs dimension. This makes it more realistic than infinitely wide networks where no feature learning occurs, but also more expressive than narrow ones or with fixed inner weights. We focus on the Bayes-optimal learning in the teacher-student scenario, i.e., with a dataset generated by another network with the same architecture. We operate around interpolation, where the number of trainable parameters and of data are comparable and feature learning emerges. Our analysis uncovers a rich phenomenology with various learning transitions as the number of data increases. In particular, the more strongly the features (i.e., hidden neurons of the target) contribute to the observed responses, the less data is needed to learn them. Moreover, when the data is scarce, the model only learns non-linear combinations of the teacher weights, rather than "specialising" by aligning its weights with the teacher's. Specialisation occurs only when enough data becomes available, but it can be hard to find for practical training algorithms, possibly due to statistical-to-computational~gaps.

This submission has been superseded by arxiv:2510.24616, which substantially improves the presentation of the theory devised here for the shallow case, adds more numerical evidence, pushes forward the approach to deeper targets/models, and has been published in Phys. Rev. X 16, 031014 (2026)

References in corpus (45)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Bilinear Generalized Approximate Message Passing
• Random matrices: Universality of local eigenvalue statistics up to the edge
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Classification and Geometry of General Perceptual Manifolds
• Statistical Mechanics of Support Vector Networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• On the Large N Limit of the Itzykson-Zuber Integral
• Analysis of CDMA systems that are characterized by eigenvalue spectrum
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Generalisation error in learning with random features and the hidden manifold model
• Rotational invariant estimator for general noisy matrices
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• High-temperature Expansions and Message Passing Algorithms
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Notes on Matrix Models
• Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
• Instanton Approach to Large $N$ Harish-Chandra-Itzykson-Zuber Integrals
• Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• Statistical learning theory of structured data
• Statistical Mechanics of Dictionary Learning
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Beyond the storage capacity: data driven satisfiability transition
• Storage of correlated patterns in a perceptron
• On the large N limit of matrix integrals over the orthogonal group
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting
• Matrix denoising: Bayes-optimal estimators via low-degree polynomials
• Random features and polynomial rules
• Coding schemes in neural networks learning classification tasks
• Spatially heterogeneous learning by a deep student machine
• Correlation of internal representations in feed-forward neural networks
• Storage capacity of correlated perceptrons
• Correlations between hidden units in multilayer neural networks and replica symmetry breaking
• Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens