11 citations · 44 across the 18 of their papers we have counts for
18 papers
Generalized Gradient Norm Clipping & Non-Euclidean -Smoothness
Thomas Pethick, Wanyun Xie, Mete Erdogan +3
This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The meth…
Layer-wise Quantization for Quantized Optimistic Dual Averaging
Anh Duc Nguyen, Ilia Markov, Frank Zhengqing Wu +4
Modern deep neural networks exhibit heterogeneity across numerous layers of various types such as residuals, multi-head attention, etc., due to varying structures (dimensions, acti…
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees
Yongtao Wu, Luca Viano, Yihang Chen +4
Reinforcement Learning from Human Feedback (RLHF) has been highly successful in aligning large language models with human preferences. While prevalent methods like DPO have demonst…
Training Deep Learning Models with Norm-Constrained LMOs
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos +3
In this work, we study optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball. We propose a new stochastic family of algorithms that uses the LMO…
Improving SAM Requires Rethinking its Optimization Formulation
Wanyun Xie, Fabian Latorre, Kimon Antonakopoulos +2
This paper rethinks Sharpness-Aware Minimization (SAM), which is originally formulated as a zero-sum game where the weights of a network and a bounded perturbation try to minimize/…
Universal Gradient Methods for Stochastic Convex Optimization
Anton Rodomanov, Ali Kavis, Yongtao Wu +2
We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the Hölder smoothness…
Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness
Artem Agafonov, Dmitry Kamzolov, Alexander Gasnikov +4
We present a new accelerated stochastic second-order method that is robust to both gradient and Hessian inexactness, which occurs typically in machine learning. We establish theore…
Distributed Extra-gradient with Optimal Complexity and Communication Guarantees
Ali Ramezani-Kebrya, Kimon Antonakopoulos, Igor Krawczuk +2
We consider monotone variational inequality (VI) problems in multi-GPU settings where multiple processors/workers/clients have access to local stochastic dual vectors. This setting…
Extra-Newton: A First Approach to Noise-Adaptive Accelerated Second-Order Methods
Kimon Antonakopoulos, Ali Kavis, Volkan Cevher
This work proposes a universal and adaptive second-order method for minimizing second-order smooth, convex functions. Our algorithm achieves convergence when the o…
Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization
Ali Kavis, Stratis Skoulakis, Kimon Antonakopoulos +2
We propose an adaptive variance-reduction method, called AdaSpider, for minimization of -smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider combines…
A universal black-box optimization method with almost dimension-free convergence rate guarantees
Kimon Antonakopoulos, Dong Quan Vu, Vokan Cevher +2
Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem's regularity parameters or the accurar…
No-Regret Learning in Games with Noisy Feedback: Faster Rates and Adaptivity via Learning Rate Separation
Yu-Guan Hsieh, Kimon Antonakopoulos, Volkan Cevher +1
We examine the problem of regret minimization when the learner is involved in a continuous game with other optimizing agents: in this case, if all players follow a no-regret algori…
Routing in an Uncertain World: Adaptivity, Efficiency, and Equilibrium
Dong Quan Vu, Kimon Antonakopoulos, Panayotis Mertikopoulos
We consider the traffic assignment problem in nonatomic routing games where the players' cost functions may be subject to random fluctuations (e.g., weather disturbances, perturbat…
Adaptive first-order methods revisited: Convex optimization without Lipschitz requirements
Kimon Antonakopoulos, Panayotis Mertikopoulos
We propose a new family of adaptive first-order methods for a class of convex minimization problems that may fail to be Lipschitz continuous or smooth in the standard sense. Specif…
Adaptive Learning in Continuous Games: Optimal Regret Bounds and Convergence to Nash Equilibrium
Yu-Guan Hsieh, Kimon Antonakopoulos, Panayotis Mertikopoulos
In game-theoretic learning, several agents are simultaneously following their individual interests, so the environment is non-stationary from each player's perspective. In this con…
On the Generalization of Stochastic Gradient Descent with Momentum
Ali Ramezani-Kebrya, Ashish Khisti, Ben Liang
While momentum-based methods, in conjunction with stochastic gradient descent (SGD), are widely used when training machine learning models, there is little theoretical understandin…
Adaptive extra-gradient methods for min-max optimization and games
Kimon Antonakopoulos, E. Veronica Belmega, Panayotis Mertikopoulos
We present a new family of min-max optimization algorithms that automatically exploit the geometry of the gradient data observed at earlier iterations to perform more informative e…
On the Generalization of Stochastic Gradient Descent with Momentum
Ali Ramezani-Kebrya, Kimon Antonakopoulos, Volkan Cevher +2
While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on t…