64 citations · 88 across the 9 of their papers we have counts for
10 papers
A Framework for Fine-Grained Synchronization of Dependent GPU Kernels
Abhinav Jangda, Saeed Maleki, Maryam Mehri Dehnavi +2
Machine Learning (ML) models execute several parallel computations including Generalized Matrix Multiplication, Convolution, Dropout, etc. These computations are commonly executed…
SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction
Zhiqi Lin, Youshan Miao, Guodong Liu +10
With the growing model size, deep neural networks (DNN) are increasingly trained over massive GPU accelerators, which demands a proper parallelization plan that transforms a DNN mo…
GC3: An Optimizing Compiler for GPU Collective Communication
Meghan Cowan, Saeed Maleki, Madanlal Musuvathi +2
Machine learning models made up of millions or billions of parameters are trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, the collec…
TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
Aashaka Shah, Vijay Chidambaram, Meghan Cowan +6
Machine learning models are increasingly being trained across multiple GPUs and servers. In this setting, data is transferred between GPUs using communication collectives such as A…
Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads
Abhinav Jangda, Jun Huang, Guodong Liu +6
Recent trend towards increasing large machine learning models require both training and inference tasks to be distributed. Considering the huge cost of training these models, it is…
Synthesizing Optimal Collective Algorithms
Zixian Cai, Zhengyang Liu, Saeed Maleki +4
Collective communication algorithms are an important component of distributed computation. Indeed, in the case of deep-learning, collective communication is the Amdahl's bottleneck…