14 citations · 30 across the 11 of their papers we have counts for
6 papers · 1 filter
Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observat…
Test like you Train in Implicit Deep Learning
Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré +1
Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks (DEQs). In its general formulation, it relies on exp…
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
Michael E. Sander, Joan Puigcerver, Josip Djolonga +2
The top-k operator returns a sparse vector, where the non-zero values correspond to the k largest values of the input. Unfortunately, because it is a discontinuous function, it is…
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Michael E. Sander, Pierre Ablin, Gabriel Peyré
Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog of Residual Neural Networks (ResNets). We investigate whether the discrete dynamics defined by a ResN…
Randomized Stochastic Gradient Descent Ascent
Othmane Sebbouh, Marco Cuturi, Gabriel Peyré
An increasing number of machine learning problems, such as robust or adversarial variants of existing algorithms, require minimizing a loss function that is itself defined as a max…
Sinkformers: Transformers with Doubly Stochastic Attention
Michael E. Sander, Pierre Ablin, Mathieu Blondel +1
Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is norm…