36 citations · 46 across the 7 of their papers we have counts for
7 papers
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction
Lei Guan, Dongsheng Li, Yongle Chen +3
Asynchronous pipeline model parallelism with a "1F1B" (one forward, one backward) schedule generates little bubble overhead and always provides quite a high throughput. However, th…
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis
Lei Guan
This paper proposes an efficient optimizer called AdaPlus which integrates Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus combines the advantages of Adam…
XGrad: Boosting Gradient-Based Optimizers With Weight Prediction
Lei Guan, Dongsheng Li, Yanqi Shi +1
In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence…
Weight Prediction Boosts the Convergence of AdamW
Lei Guan
In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models. In particular, ahead of each min…
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
Lei Guan, Wotao Yin, Dongsheng Li +1
We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training. XPipe is designed to use multiple GPUs to concurrently and continuously…
Non-ergodic Convergence Analysis of Heavy-Ball Algorithms
Tao Sun, Penghang Yin, Dongsheng Li +3
In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/…