Lenka Zdeborová

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

Haotian Gu, Yizhou Xu, Lenka Zdeborová

In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target…

High-Dimensional Learning Dynamics of Attention-Indexed Models

Yizhou Xu, Margarita Sagitova, Lenka Zdeborová +1

Attention mechanisms are central to modern foundation models, yet their training dynamics remain poorly understood, especially when the attention matrices have extensive rank. In t…

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

Julius Girardin, Emanuele Troiani, Yizhou Xu +3

Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe…

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

O. Duranthon, F. Boncoraglio, L. Zdeborová

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a sol…

Specialization of softmax attention heads: insights from the high-dimensional single-location model

M. Sagitova, O. Duranthon, L. Zdeborová

Multi-head attention enables transformer models to represent multiple attention patterns simultaneously. Empirically, head specialization emerges in distinct stages during training…

Inference in Spreading Processes with Neural-Network Priors

Davide Ghio, Fabrizio Boncoraglio, Lenka Zdeborová

Stochastic processes on graphs are a powerful tool for modelling complex dynamical systems such as epidemics. A recent line of work focused on the inference problem where one aims…

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani +3

Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the…

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime

Leonardo Defilippis, Yizhou Xu, Julius Girardin +5

Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a…

Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge

Freya Behrens, Lenka Zdeborová

Dataset distillation aims to compress training data into fewer examples via a teacher, from which a student can learn effectively. While its success is often attributed to structur…

cond-mat.dis-nn2025

Sequential Dynamics in Ising Spin Glasses

Yatin Dandi, David Gamarnik, Francisco Pernice +1

We present the first exact asymptotic characterization of sequential dynamics for a broad class of local update algorithms on the Sherrington-Kirkpatrick (SK) model with Ising spin…

cond-mat.dis-nn2025

The maximum-average subtensor problem: equilibrium and out-of-equilibrium properties

Vittorio Erba, Nathan Malo Kupferschmid, Rodrigo Pérez Ortiz +1

In this paper we introduce and study the Maximum-Average Subtensor ( $p$ -MAS) problem, in which one wants to find a subtensor of size $k$ of a given random tensor of size $N$ , both…

On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification

Matteo Vilucchio, Lenka Zdeborová, Bruno Loureiro

What fundamentally distinguishes an adversarial attack from a misclassification due to limited model expressivity or finite data? In this work, we investigate this question in the…

Bayes optimal learning of attention-indexed models

Fabrizio Boncoraglio, Emanuele Troiani, Vittorio Erba +1

We introduce the attention-indexed model (AIM), a theoretical framework for analyzing learning in deep attention layers. Inspired by multi-index models, AIM captures how token-leve…

Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions

Yizhou Xu, Florent Krzakala, Lenka Zdeborová

The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its perfor…

The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks

Vittorio Erba, Emanuele Troiani, Lenka Zdeborová +1

We study the high-dimensional asymptotics of empirical risk minimization (ERM) in over-parametrized two-layer neural networks with quadratic activations trained on synthetic data.…

Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applications

Yizhou Xu, Antoine Maillard, Lenka Zdeborová +1

In the matrix sensing problem, one wishes to reconstruct a matrix from (possibly noisy) observations of its linear projections along given directions. We consider this model in the…