63 citations · 137 across the 16 of their papers we have counts for
8 papers · 1 filter
Convergence of Rotation-based Matrix Optimizers: A Unified Analysis of SOAP, Conda, and SPlus
Yiwen Sun, Huan Li, Zhouchen Lin
In this work, we develop a unified theoretical framework for analyzing the convergence of rotation-based matrix optimizers, which apply orthogonal transformations to map the moment…
Convergence Rate Analysis of SOAP with Arbitrary Orthogonal Projection Matrices
Huan Li, Zhouchen Lin
In this short note, we establish, for the first time, the convergence rate of SOAP, an efficient and popular matrix-based optimizer for training deep neural networks. Our analysis…
Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning
Huan Li, Yiming Dong, Zhouchen Lin
This paper studies AdamW-style Shampoo, an effective variant of the classical Shampoo that won the external tuning track of the AlgoPerf neural network training competition. Our an…
On the Convergence Rate of RMSProp and Its Momentum Extension Measured by Norm
Huan Li, Yiming Dong, Zhouchen Lin
Although adaptive gradient methods have been extensively used in deep learning, their convergence rates proved in the literature are all slower than that of SGD, particularly with…
Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the Complexity
Huan Li, Zhouchen Lin
This paper studies accelerated gradient methods for nonconvex optimization with Lipschitz continuous gradient and Hessian. We propose two simple accelerated gradient methods, resta…
Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization
Huan Li, Zhouchen Lin
Decentralized optimization over time-varying graphs has been increasingly common in modern machine learning with massive data stored on millions of mobile devices, such as in feder…
Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization
Huan Li, Zhouchen Lin, Yongchun Fang
We study stochastic decentralized optimization for the problem of training machine learning models with large-scale distributed data. We extend the widely used EXTRA and DIGing met…
Decentralized Accelerated Gradient Methods With Increasing Penalty Parameters
Huan Li, Cong Fang, Wotao Yin +1
In this paper, we study the communication and (sub)gradient computation costs in distributed optimization and give a sharp complexity analysis for the proposed distributed accelera…