12 papers
Direct Acceleration of Stochastic Root-Finding Without Variance Reduction and Regularization
TaeHo Yoon, Nicolas Loizou
Acceleration for deterministic root-finding problems has been extensively studied in recent years; specifically, the anchor-based, or Halpern-type methods achieve optimal convergen…
On Same-Sample and Independent-Sample Stochastic Extragradient for Monotone Variational Inequalities
TaeHo Yoon, Nicolas Loizou
We study stochastic extragradient (SEG) methods for solving monotone variational inequality problems (VIPs) over a feasible set. Although extragradient is a foundational algorithm…
Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
Dimitris Oikonomou, Nicolas Loizou
Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models. By explicitly minimizing the sharpness of…
Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients
Dimitris Oikonomou, Nicolas Loizou
The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art met…
Certified Robustness from Approximate Gaussian Mixture Structures in Pretrained Latent Spaces
Konstantinos Emmanouilidis, Tianjiao Ding, Nghia Nguyen +2
Deep learning models are vulnerable to adversarial perturbations, raising important concerns for safety-critical deployment. Empirical defenses can achieve strong robustness in pra…
Taking the Road Less Scheduled with Adaptive Polyak Steps
Dimitris Oikonomou, Matthew Buchholz, Yuen-Man Pun +2
Schedule-Free SGD, proposed in [Defazio et al., 2024], achieves optimal convergence rates without requiring the training horizon in advance, by replacing learning rate schedules wi…