4 papers · 1 filter
Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients
Dimitris Oikonomou, Nicolas Loizou
The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art met…
Taking the Road Less Scheduled with Adaptive Polyak Steps
Dimitris Oikonomou, Matthew Buchholz, Yuen-Man Pun +2
Schedule-Free SGD, proposed in [Defazio et al., 2024], achieves optimal convergence rates without requiring the training horizon in advance, by replacing learning rate schedules wi…
Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation
Robert M. Gower, Guillaume Garrigos, Nicolas Loizou +3
We provide a general convergence theorem of an idealized stochastic Polyak step size called SPS. Besides convexity, we only assume a local expected gradient bound, that include…
Sharpness-Aware Minimization: General Analysis and Improved Rates
Dimitris Oikonomou, Nicolas Loizou
Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape. Howev…