82 citations · 360 across the 42 of their papers we have counts for
43 papers
Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2
We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-de…
Optimal scaling laws in learning hierarchical multi-index models
Leonardo Defilippis, Florent Krzakala, Bruno Loureiro +1
In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited…
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
Leonardo Defilippis, Yizhou Xu, Julius Girardin +5
Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a…
Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks
Luca Arnaboldi, Bruno Loureiro, Ludovic Stephan +2
We study the dynamics of stochastic gradient descent (SGD) for a class of sequence models termed Sequence Single-Index (SSI) models, where the target depends on a single direction…
Optimal Spectral Transitions in High-Dimensional Multi-Index Models
Leonardo Defilippis, Yatin Dandi, Pierre Mergny +2
We consider the problem of how many samples from a Gaussian multi-index model are required to weakly reconstruct the relevant index subspace. Despite its increasing popularity as a…
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities
Yatin Dandi, Luca Pesce, Hugo Cui +3
A key property of neural networks is their capacity of adapting to data during training. Yet, our current mathematical understanding of feature learning and its relationship to gen…
Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs
Luca Arnaboldi, Yatin Dandi, Florent Krzakala +3
We study the impact of the batch size on the iteration time of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target fu…
Dimension-free deterministic equivalents and scaling laws for random feature regression
Leonardo Defilippis, Bruno Loureiro, Theodor Misiakiewicz
In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error…
Fundamental computational limits of weak learnability in high-dimensional multi-index models
Emanuele Troiani, Yatin Dandi, Leonardo Defilippis +3
Multi-index models - functions which only depend on the covariates through a non-linear transformation of their projection on a subspace - are a useful benchmark for investigating…
Asymptotics of Learning with Deep Structured (Random) Features
Dominik Schröder, Daniil Dmitriev, Hugo Cui +1
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where t…
Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression
Lucas Clarté, Adrien Vandenbroucque, Guillaume Dalle +3
We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dime…
A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs
Kasimir Tanner, Matteo Vilucchio, Bruno Loureiro +1
This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension and the number of data points $…
Asymptotics of feature learning in two-layer networks after one gradient-step
Hugo Cui, Luca Pesce, Yatin Dandi +4
In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single grad…
High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality
Urte Adomaityte, Leonardo Defilippis, Bruno Loureiro +1
We investigate the high-dimensional properties of robust regression estimators in the presence of heavy-tailed contamination of both the covariates and response functions. In parti…
Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD
Luca Arnaboldi, Florent Krzakala, Bruno Loureiro +1
This study explores the sample complexity for two-layer neural networks to learn a generalized linear target function under Stochastic Gradient Descent (SGD), focusing on the chall…
How Two-Layer Neural Networks Learn, One (Giant) Step at a Time
Yatin Dandi, Florent Krzakala, Bruno Loureiro +2
For high-dimensional Gaussian data, we investigate theoretically how the features of a two-layer neural network adapt to the structure of the target function through a few large ba…
Expectation consistency for calibration of neural networks
Lucas Clarté, Bruno Loureiro, Florent Krzakala +1
Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence. Finding effective and efficient cal…
Universality laws for Gaussian mixtures in generalized linear models
Yatin Dandi, Ludovic Stephan, Florent Krzakala +2
Let denote independent samples from a general mixture distribution , and consider the hypothesis class of gener…
Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation
Luca Pesce, Florent Krzakala, Bruno Loureiro +1
In this manuscript we consider the problem of generalized linear estimation on Gaussian mixture data with labels given by a single-index model. Our first result is a sharp asymptot…
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks
Luca Arnaboldi, Ludovic Stephan, Florent Krzakala +1
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a two-layer neural network trained on Gaussian data and labels generated by a similar, thoug…
Deterministic equivalent and error universality of deep random features learning
Dominik Schröder, Hugo Cui, Daniil Dmitriev +1
This manuscript considers the problem of learning a random Gaussian network function using a fully connected network with frozen intermediate layers and trainable readout layer. Th…
On double-descent in uncertainty quantification in overparametrized models
Lucas Clarté, Bruno Loureiro, Florent Krzakala +1
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estim…
Secure Coding via Gaussian Random Fields
Ali Bereyhi, Bruno Loureiro, Florent Krzakala +2
Inverse probability problems whose generative models are given by strictly nonlinear Gaussian random fields show the all-or-nothing behavior: There exists a critical rate at which…
Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
Luca Pesce, Bruno Loureiro, Florent Krzakala +1
A simple model to study subspace clustering is the high-dimensional -Gaussian mixture model where the cluster means are sparse vectors. Here we provide an exact asymptotic chara…