1 citations · 1 across the 7 of their papers we have counts for
4 papers · 1 filter
Convergence of optimizers implies eigenvalues filtering at equilibrium
Jerome Bolte, Quoc-Tung Le, Edouard Pauwels
Ample empirical evidence in deep neural network training suggests that a variety of optimizers tend to find nearly global optima. In this article, we adopt the reversed perspective…
When majority rules, minority loses: bias amplification of gradient descent
François Bachoc, Jérôme Bolte, Ryan Boustany +1
Despite growing empirical evidence of bias amplification in machine learning, its theoretical foundations remain poorly understood. We develop a formal framework for majority-minor…
A second-order-like optimizer with adaptive gradient scaling for deep learning
Jérôme Bolte, Ryan Boustany, Edouard Pauwels +1
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order info…
Numerical influence of ReLU'(0) on backpropagation
David Bertoin, Jérôme Bolte, Sébastien Gerchinovitz +1
In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training. Yet, in the real world, 32 bits default precision c…