51 citations · 209 across the 15 of their papers we have counts for
3 papers · 1 filter
Stochastic Gradient Descent without Full Data Shuffle
Lijie Xu, Shuang Qiu, Binhang Yuan +9
Stochastic gradient descent (SGD) is the cornerstone of modern machine learning (ML) systems. Despite its computational efficiency, SGD requires random data access that is inherent…
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Jue Wang, Binhang Yuan, Luka Rimanic +5
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive…
Decentralized Training of Foundation Models in Heterogeneous Environments
Binhang Yuan, Yongjun He, Jared Quincy Davis +6
Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months. These models are typicall…