2 citations · 5 across the 16 of their papers we have counts for
16 papers
Transformative or Conservative? Conservation laws for ResNets and Transformers
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practi…
Learning from samples: inverse problems over measures
Francisco Andrade, Gabriel Peyré, Clarice Poon
We study inverse problems where an unknown potential is observed only through samples from the measure it induces by a convex variational principle. Such problems arise in learning…
Optimal Transport for Machine Learners
Gabriel Peyré
Modern machine learning repeatedly manipulates probability measures: empirical datasets, generated samples, latent distributions, class-conditional laws, particle systems, weights…
Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime
Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard
We study the convergence of gradient methods for the training of mean-field single-hidden-layer neural networks with square loss. For this high-dimensional and non-convex optimizat…
From Score Matching to Diffusion: A Fine-Grained Error Analysis in the Gaussian Setting
Samuel Hurault, Matthieu Terris, Thomas Moreau +1
Sampling from an unknown distribution, accessible only through discrete samples, is a fundamental problem at the core of generative AI. The current state-of-the-art methods follow…
The Mathematics of Artificial Intelligence
Gabriel Peyré
This overview article highlights the critical role of mathematics in artificial intelligence (AI), emphasizing that mathematics provides tools to better understand and enhance AI s…
A Unified Perspective on the Dynamics of Deep Transformers
Valérie Castin, Pierre Ablin, José Antonio Carrillo +1
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the atte…
Towards Understanding the Universality of Transformers for Next-Token Prediction
Michael E. Sander, Gabriel Peyré
Causal Transformers are trained to predict the next token for a given context. While it is widely accepted that self-attention is crucial for encoding the causal structure of seque…
Transformers are Universal In-context Learners
Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
Transformers are deep architectures that define "in-context mappings" which enable predicting new tokens based on a given set of tokens (such as a prompt in NLP applications or a s…
Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles f…
Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport
Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard
We study the convergence of gradient flow for the training of deep neural networks. If Residual Neural Networks are a popular example of very deep architectures, their training con…
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
Zhenzhang Ye, Gabriel Peyré, Daniel Cremers +1
Bilevel optimization aims to optimize an outer objective function that depends on the solution to an inner optimization problem. It is routinely used in Machine Learning, notably f…
How do Transformers perform In-Context Autoregressive Learning?
Michael E. Sander, Raja Giryes, Taiji Suzuki +2
Transformers have achieved state-of-the-art performance in language modeling tasks. However, the reasons behind their tremendous success are still unclear. In this paper, towards a…
How Smooth Is Attention?
Valérie Castin, Pierre Ablin, Gabriel Peyré
Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz p…
Structured Transforms Across Spaces with Cost-Regularized Optimal Transport
Othmane Sebbouh, Marco Cuturi, Gabriel Peyré
Matching a source to a target probability measure is often solved by instantiating a linear optimal transport (OT) problem, parameterized by a ground cost function that quantifies…
Super-resolved Lasso
Clarice Poon, Gabriel Peyré
Super-resolution of pointwise sources is of utmost importance in various areas of imaging sciences. Specific instances of this problem arise in single molecule fluorescence, spike…