5.2k citations · 7.8k across the 38 of their papers we have counts for
14 papers · 1 filter
Birth of a Transformer: A Memory Viewpoint
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt +2
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their i…
A Simple Convergence Proof of Adam and Adagrad
Alexandre Défossez, Léon Bottou, Francis Bach +1
We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with b…
Invariant Risk Minimization
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani +1
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a da…
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
Rachel Ward, Xiaoxia Wu, Leon Bottou
Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such met…
WNGrad: Learn the Learning Rate in Gradient Descent
Xiaoxia Wu, Rachel Ward, Léon Bottou
Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function…
First-order Adversarial Vulnerability of Neural Networks and Input Dimension
Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou +2
Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We…