Differentiable Self-Adaptive Learning Rate

Bozhou Chen, Hongzhi Wang, Chenmin Ba

Learning rate adaptation is a popular topic in machine learning. Gradient Descent trains neural nerwork with a fixed learning rate. Learning rate adaptation is proposed to accelera…