No More Adam: Learning Rate Scaling at Initialization is All You Need

Minghao Xu, Lichuan Xiang, Xu Cai +1

In this work, we question the necessity of adaptive gradient methods for training deep neural networks. SGD-SaI is a simple yet effective enhancement to stochastic gradient descent…