Read the Signs: Towards Invariance to Gradient Descent's Hyperparameter Initialization

Davood Wadi, Marc Fredette, Sylvain Senecal

We propose ActiveLR, an optimization meta algorithm that localizes the learning rate, $\alpha$, and adapts them at each epoch according to whether the gradient at each epoch changes sig…