63.6k citations · 64.4k across the 30 of their papers we have counts for
30 papers
Dual Gauss-Newton Directions for Deep Learning
Vincent Roulet, Mathieu Blondel
Inspired by Gauss-Newton-like methods, we study the benefit of leveraging the structure of deep learning objectives, namely, the composition of a convex loss function and of a nonl…
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
Michael E. Sander, Joan Puigcerver, Josip Djolonga +2
The top-k operator returns a sparse vector, where the non-zero values correspond to the k largest values of the input. Unfortunately, because it is a discontinuous function, it is…
Sparsity-Constrained Optimal Transport
Tianlin Liu, Joan Puigcerver, Mathieu Blondel
Regularized optimal transport (OT) is now increasingly used as a loss or as a matching layer in neural networks. Entropy-regularized OT can be computed using the Sinkhorn algorithm…
Learning Energy Networks with Generalized Fenchel-Young Losses
Mathieu Blondel, Felipe Llinares-López, Robert Dadashi +2
Energy-based models, a.k.a. energy networks, perform inference by optimizing an energy function, typically parametrized by a neural network. This allows one to capture potentially…
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
Robert M. Gower, Mathieu Blondel, Nidham Gazagnadou +1
Tuning the step size of stochastic gradient descent is tedious and error prone. This has motivated the development of methods that automatically adapt the step size using readily a…
Sinkformers: Transformers with Doubly Stochastic Attention
Michael E. Sander, Pierre Ablin, Mathieu Blondel +1
Attention based models such as Transformers involve pairwise interactions between data points, modeled with a learnable attention matrix. Importantly, this attention matrix is norm…