4 papers
Revisiting AdaGrad in Stochastic Convex Optimization: Last Iterates, High Probability, and Lower Bounds
Weiming Ou, Xiao Wang
AdaGrad and AdaGrad-Norm are widely used adaptive methods, but their precise behavior in stochastic convex optimization remains less understood. We first show that AdaGrad-Norm and…
Tight Sampling Complexity with Stochastic Gradient Oracles in Fixed Dimensions
Weiming Ou, Xiao Wang
We establish the stochastic-gradient query complexity of sampling smooth strongly log-concave distributions in every fixed dimension . For -strongly convex, -smooth p…
Tight Heavy-Tailed Complexity Bounds for Optimizing Polyak--Łojasiewicz Objectives
Weiming Ou, Xiao Wang
We study optimizing -smooth and -Polyak-Lojasiewicz (PL) objectives with unbiased stochastic gradients under -heavy-tailed noise (). In the unrestricted high-dim…
Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach
Yi Feng, Weiming Ou, Xiao Wang
The remarkable success of the Adam in training neural networks has naturally led to the widespread use of its descent-ascent counterpart, Adam-DA, for solving zero-sum games. Despi…