885 citations · 2.1k across the 60 of their papers we have counts for
4 papers · 2 filters
The Effect of Network Width on the Performance of Large-batch Training
Lingjiao Chen, Hongyi Wang, Jinman Zhao +2
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in sm…
ATOMO: Communication-efficient Learning via Atomic Sparsification
Hongyi Wang, Scott Sievert, Zachary Charles +3
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies pro…
Gradient Coding via the Stochastic Block Model
Zachary Charles, Dimitris Papailiopoulos
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size an…
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Lingjiao Chen, Hongyi Wang, Zachary Charles +1
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a…