9 papers
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Julius Girardin, Emanuele Troiani, Yizhou Xu +3
Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe…
Percolation and clustering in ecological communities: A dynamical theory
Dario Sergo, Cédric Koller, Vittorio Erba +1
Ecological communities with structured interactions exhibit collective phenomena such as percolation and clustering of occupied sites. While these effects have been documented in e…
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Leonardo Defilippis, Yizhou Xu, Julius Girardin +6
Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a…
A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalization
Vicente Conde Mendes, Lorenzo Bardone, Cédric Koller +6
Many real-world datasets contain hidden structure that cannot be detected by simple linear correlations between input features. For example, latent factors may influence the data i…
Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani +3
Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the…
Bayes optimal learning of attention-indexed models
Fabrizio Boncoraglio, Emanuele Troiani, Vittorio Erba +2
We introduce the attention-indexed model (AIM), a theoretical framework for analyzing learning in deep attention layers. Inspired by multi-index models, AIM captures how token-leve…