SING: A Plug-and-Play DNN Learning Technique

Adrien Courtois, Damien Scieur, Jean-Michel Morel +2

We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to…