36 citations · 46 across the 7 of their papers we have counts for
4 papers · 1 filter
PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction
Lei Guan, Dongsheng Li, Yongle Chen +3
Asynchronous pipeline model parallelism with a "1F1B" (one forward, one backward) schedule generates little bubble overhead and always provides quite a high throughput. However, th…
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis
Lei Guan
This paper proposes an efficient optimizer called AdaPlus which integrates Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus combines the advantages of Adam…
XGrad: Boosting Gradient-Based Optimizers With Weight Prediction
Lei Guan, Dongsheng Li, Yanqi Shi +1
In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence…
Weight Prediction Boosts the Convergence of AdamW
Lei Guan
In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models. In particular, ahead of each min…