Step by Step: Adaptive Gradient Descent for Training L-Lipschitz Neural Networks

Kyle Sung, Kholood Khalil, Noah Forman +2

We demonstrate that applying an eventual decay to the learning rate (LR) in empirical risk minimization (ERM), where the mean-squared-error loss is minimized using standard gradien…