67 citations · 73 across the 2 of their papers we have counts for
2 papers
cs.LG2022★ 67 cited
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Zhiquan Lai, Shengwei Li, Xudong Tang +5
Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter an…
cs.LG2021★ 6 cited
EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
Shengwei Li, Zhiquan Lai, Dongsheng Li +3
Distributed data-parallel training has been widely adopted for deep neural network (DNN) models. Although current deep learning (DL) frameworks scale well for dense models like ima…