167 citations · 446 across the 8 of their papers we have counts for
14 papers
AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness
Dacheng Li, Hongyi Wang, Eric Xing +1
Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks. However, training big models requires strong distributed system expertise to…
Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
Lingjiao Chen, Leshang Chen, Hongyi Wang +2
There has been a growing need to provide Byzantine-resilience in distributed model training. Existing robust distributed learning algorithms focus on developing sophisticated robus…
A Field Guide to Federated Optimization
Jianyu Wang, Zachary Charles, Zheng Xu +50
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy prote…
Pufferfish: Communication-efficient Models At No Extra Cost
Hongyi Wang, Saurabh Agarwal, Dimitris Papailiopoulos
To mitigate communication overheads in distributed model training, several studies propose the use of compressed stochastic gradients, usually achieved by sparsification or quantiz…
On the Utility of Gradient Compression in Distributed Training Systems
Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman +1
A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent wo…
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
Saurabh Agarwal, Hongyi Wang, Kangwook Lee +2
Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use…