Showing 2024Show all
2 papers · 1 filter
cs.LG2024
A second-order-like optimizer with adaptive gradient scaling for deep learning
Jérôme Bolte, Ryan Boustany, Edouard Pauwels +1
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order info…
cs.LG2024
On the numerical reliability of nonsmooth autodiff: a MaxPool case study
Ryan Boustany
This paper considers the reliability of automatic differentiation (AD) for neural networks involving the nonsmooth MaxPool operation. We investigate the behavior of AD across diffe…