6 papers
Random Reshuffling Dominates Stochastic Gradient Descent
Zijian Liu
Stochastic Gradient Descent () is one of the most classical optimization algorithms with favorable theoretical guarantees, yet the practical implementation of $\texts…
Improved Last-Iterate Convergence of Shuffling Gradient Methods for Nonsmooth Convex Optimization
Zijian Liu, Zhengyuan Zhou
We study the convergence of the shuffling gradient method, a popular algorithm employed to minimize the finite-sum function with regularization, in which functions are passed to ap…
Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping
Zijian Liu, Zhengyuan Zhou
Recently, the study of heavy-tailed noises in first-order nonconvex stochastic optimization has gotten a lot of attention since it was recognized as a more realistic condition as s…
On the Last-Iterate Convergence of Shuffling Gradient Methods
Zijian Liu, Zhengyuan Zhou
Shuffling gradient methods are widely used in modern machine learning tasks and include three popular implementations: Random Reshuffle (RR), Shuffle Once (SO), and Incremental Gra…
Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods
Zijian Liu, Zhengyuan Zhou
In the past several years, the last-iterate convergence of the Stochastic Gradient Descent (SGD) algorithm has triggered people's interest due to its good performance in practice b…
META-STORM: Generalized Fully-Adaptive Variance Reduced SGD for Unbounded Functions
Zijian Liu, Ta Duy Nguyen, Thien Hang Nguyen +2
We study the application of variance reduction (VR) techniques to general non-convex stochastic optimization problems. In this setting, the recent work STORM [Cutkosky-Orabona '19]…