5.2k citations · 7.8k across the 38 of their papers we have counts for
6 papers · 1 filter
Controlling Covariate Shift using Balanced Normalization of Weights
Aaron Defazio, Léon Bottou
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. Th…
On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
Aaron Defazio, Léon Bottou
The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard no…
SING: Symbol-to-Instrument Neural Generator
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier +2
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders…
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
Rachel Ward, Xiaoxia Wu, Leon Bottou
Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such met…
WNGrad: Learn the Learning Rate in Gradient Descent
Xiaoxia Wu, Rachel Ward, Léon Bottou
Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function…
First-order Adversarial Vulnerability of Neural Networks and Input Dimension
Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou +2
Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We…