30 citations · 112 across the 22 of their papers we have counts for
5 papers · 1 filter
Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani +3
Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the…
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Leonardo Defilippis, Yizhou Xu, Julius Girardin +5
Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a…
The maximum-average subtensor problem: equilibrium and out-of-equilibrium properties
Vittorio Erba, Nathan Malo Kupferschmid, Rodrigo Pérez Ortiz +1
In this paper we introduce and study the Maximum-Average Subtensor (-MAS) problem, in which one wants to find a subtensor of size of a given random tensor of size , both…
Bayes optimal learning of attention-indexed models
Fabrizio Boncoraglio, Emanuele Troiani, Vittorio Erba +1
We introduce the attention-indexed model (AIM), a theoretical framework for analyzing learning in deep attention layers. Inspired by multi-index models, AIM captures how token-leve…
The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks
Vittorio Erba, Emanuele Troiani, Lenka Zdeborová +1
We study the high-dimensional asymptotics of empirical risk minimization (ERM) in over-parametrized two-layer neural networks with quadratic activations trained on synthetic data.…