885 citations · 2.1k across the 60 of their papers we have counts for
11 papers · 1 filter
The Effect of Network Width on the Performance of Large-batch Training
Lingjiao Chen, Hongyi Wang, Jinman Zhao +2
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in sm…
ATOMO: Communication-efficient Learning via Atomic Sparsification
Hongyi Wang, Scott Sievert, Zachary Charles +3
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies pro…
Gradient Coding via the Stochastic Block Model
Zachary Charles, Dimitris Papailiopoulos
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size an…
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients
Lingjiao Chen, Hongyi Wang, Zachary Charles +1
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a…
Approximate Gradient Coding via Sparse Random Graphs
Zachary Charles, Dimitris Papailiopoulos, Jordan Ellenberg
Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have bee…
Stability and Generalization of Learning Algorithms that Converge to Global Optima
Zachary Charles, Dimitris Papailiopoulos
We establish novel generalization bounds for learning algorithms that converge to global minima. We do so by deriving black-box stability results that only depend on the convergenc…