Online Learning-guided Learning Rate Adaptation via Gradient Alignment

Ruichen Jiang, Ali Kavis, Aryan Mokhtari

The performance of an optimizer on large-scale deep learning models depends critically on fine-tuning the learning rate, often requiring an extensive grid search over base learning…