263 citations · 392 across the 53 of their papers we have counts for
9 papers · 1 filter
How Smooth Is Attention?
Valérie Castin, Pierre Ablin, Gabriel Peyré
Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz p…
Adaptive Training Distributions with Scalable Online Bilevel Optimization
David Grangier, Pierre Ablin, Awni Hannun
Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely…
A Challenge in Reweighting Data with Bilevel Optimization
Anastasia Ivanova, Pierre Ablin
In many scenarios, one uses a large training set to train a model with the goal of performing well on a smaller testing set with a different distribution. Learning a weight for eac…
How to Scale Your EMA
Dan Busbridge, Jason Ramapuram, Pierre Ablin +4
Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-o…
Learning Elastic Costs to Shape Monge Displacements
Michal Klein, Aram-Alexandre Pooladian, Pierre Ablin +3
Given a source and a target probability measure supported on , the Monge problem asks to find the most efficient way to map one distribution to the other. This effici…
Test like you Train in Implicit Deep Learning
Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré +1
Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks (DEQs). In its general formulation, it relies on exp…