2 citations · 5 across the 16 of their papers we have counts for
10 papers · 1 filter
Transformative or Conservative? Conservation laws for ResNets and Transformers
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practi…
Learning from samples: inverse problems over measures
Francisco Andrade, Gabriel Peyré, Clarice Poon
We study inverse problems where an unknown potential is observed only through samples from the measure it induces by a convex variational principle. Such problems arise in learning…
Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime
Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard
We study the convergence of gradient methods for the training of mean-field single-hidden-layer neural networks with square loss. For this high-dimensional and non-convex optimizat…
From Score Matching to Diffusion: A Fine-Grained Error Analysis in the Gaussian Setting
Samuel Hurault, Matthieu Terris, Thomas Moreau +1
Sampling from an unknown distribution, accessible only through discrete samples, is a fundamental problem at the core of generative AI. The current state-of-the-art methods follow…
A Unified Perspective on the Dynamics of Deep Transformers
Valérie Castin, Pierre Ablin, José Antonio Carrillo +1
Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens. This representation is then exploited by the atte…
Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles f…
Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport
Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard
We study the convergence of gradient flow for the training of deep neural networks. If Residual Neural Networks are a popular example of very deep architectures, their training con…
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
Zhenzhang Ye, Gabriel Peyré, Daniel Cremers +1
Bilevel optimization aims to optimize an outer objective function that depends on the solution to an inner optimization problem. It is routinely used in Machine Learning, notably f…
How Smooth Is Attention?
Valérie Castin, Pierre Ablin, Gabriel Peyré
Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz p…
Structured Transforms Across Spaces with Cost-Regularized Optimal Transport
Othmane Sebbouh, Marco Cuturi, Gabriel Peyré
Matching a source to a target probability measure is often solved by instantiating a linear optimal transport (OT) problem, parameterized by a ground cost function that quantifies…