4 papers
When majority rules, minority loses: bias amplification of gradient descent
François Bachoc, Jérôme Bolte, Ryan Boustany +1
Despite growing empirical evidence of bias amplification in machine learning, its theoretical foundations remain poorly understood. We develop a formal framework for majority-minor…
A second-order-like optimizer with adaptive gradient scaling for deep learning
Jérôme Bolte, Ryan Boustany, Edouard Pauwels +1
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order info…
On the numerical reliability of nonsmooth autodiff: a MaxPool case study
Ryan Boustany
This paper considers the reliability of automatic differentiation (AD) for neural networks involving the nonsmooth MaxPool operation. We investigate the behavior of AD across diffe…
On the complexity of nonsmooth automatic differentiation
Jérôme Bolte, Ryan Boustany, Edouard Pauwels +1
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide…