Adaptive scaling of the learning rate by second order automatic differentiation

Frédéric de Gournay, Alban Gossard

In the context of the optimization of Deep Neural Networks, we propose to rescale the learning rate using a new technique of automatic differentiation. This technique relies on the…