Demon: Improved Neural Network Training with Momentum Decay

John Chen, Cameron Wolfe, Zhao Li +1

Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule. We conduct the first large-…