kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networks

Tao Zhang, Wei Li

Recent work has shown that optimizing the Learning Rate (LR) schedule can be a very accurate and efficient way to train deep neural networks. We observe that the rate of change (RO…