67 citations · 76 across the 5 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2023★ 2 cited
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Xiaoge Deng, Li Shen, Shengwei Li +3
Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance…
cs.LG2022★ 67 cited
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Zhiquan Lai, Shengwei Li, Xudong Tang +5
Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter an…
cs.LG2021★ 6 cited
EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
Shengwei Li, Zhiquan Lai, Dongsheng Li +3
Distributed data-parallel training has been widely adopted for deep neural network (DNN) models. Although current deep learning (DL) frameworks scale well for dense models like ima…