110 citations · 230 across the 12 of their papers we have counts for
4 papers · 1 filter
TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
Aashaka Shah, Vijay Chidambaram, Meghan Cowan +6
Machine learning models are increasingly being trained across multiple GPUs and servers. In this setting, data is transferred between GPUs using communication collectives such as A…
Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads
Abhinav Jangda, Jun Huang, Guodong Liu +6
Recent trend towards increasing large machine learning models require both training and inference tasks to be distributed. Considering the huge cost of training these models, it is…
Synthesizing Optimal Collective Algorithms
Zixian Cai, Zhengyang Liu, Saeed Maleki +4
Collective communication algorithms are an important component of distributed computation. Indeed, in the case of deep-learning, collective communication is the Amdahl's bottleneck…
Scaling Distributed Training with Adaptive Summation
Saeed Maleki, Madan Musuvathi, Todd Mytkowicz +5
Stochastic gradient descent (SGD) is an inherently sequential training algorithm--computing the gradient at batch depends on the model parameters learned from batch . Prio…