Taking the Road Less Scheduled with Adaptive Polyak Steps

Dimitris Oikonomou, Matthew Buchholz, Yuen-Man Pun +2

Schedule-Free SGD, proposed in [Defazio et al., 2024], achieves optimal convergence rates without requiring the training horizon in advance, by replacing learning rate schedules wi…