Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation

Abstract

We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms.

This submission has been superseded by arxiv:2510.24616, which describes a unified theory to derive the phase diagram of the model, accounts for non-homogeneous readouts, pushes forward the approach to deeper targets/models, and has been published in Phys. Rev. X 16, 031014 (2026)

References in corpus (39)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Bilinear Generalized Approximate Message Passing
• Random matrices: Universality of local eigenvalue statistics up to the edge
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Classification and Geometry of General Perceptual Manifolds
• Statistical Mechanics of Support Vector Networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Analysis of CDMA systems that are characterized by eigenvalue spectrum
• Generalisation error in learning with random features and the hidden manifold model
• Rotational invariant estimator for general noisy matrices
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• High-temperature Expansions and Message Passing Algorithms
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Notes on Matrix Models
• Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
• Instanton Approach to Large $N$ Harish-Chandra-Itzykson-Zuber Integrals
• Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• Statistical learning theory of structured data
• Statistical Mechanics of Dictionary Learning
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Beyond the storage capacity: data driven satisfiability transition
• On the large N limit of matrix integrals over the orthogonal group
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting
• Matrix denoising: Bayes-optimal estimators via low-degree polynomials
• Random features and polynomial rules
• Coding schemes in neural networks learning classification tasks
• Spatially heterogeneous learning by a deep student machine
• Correlations between hidden units in multilayer neural networks and replica symmetry breaking
• Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens