Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

Yanzhao Wu, Ling Liu, Juhyun Bae +6

Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivi…