25 papers
Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms
Lannie Dalton Hough, Emir Gencer, Hoffmann Muki +1
High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-the-art collectiv…
Understanding and Improving Communication Performance in Multi-node LLM Inference
Prajwal Singhania, Siddharth Singh, Lannie Dalton Hough +4
As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only acr…
KEET: Explaining Performance of GPU Kernels Using LLM Agents
Joshua H. Davis, Klaudiusz Rydzy, Srinivasan Ramesh +5
Performance profiles of GPU kernels generated by tools such as Nsight Compute are rich in detail but are often challenging to interpret. To achieve the best performance possible on…
Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training
Cunyang Wei, Siddharth Singh, Aishwarya Sarkar +7
Graph neural networks (GNNs) are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed tra…
The Big Send-off: Scalable and Performant Collectives for Deep Learning
Siddharth Singh, Keshav Pradeep, Mahua Singh +2
Collective communication is becoming increasingly important in data center and supercomputer workloads with an increase in distributed AI related jobs. However, existing libraries…
Speculating Experts Accelerates Inference for Mixture-of-Experts
Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token…