55 citations · 156 across the 9 of their papers we have counts for
3 papers · 1 filter
NCCL M2N: A Layout- and Topology-Aware Collective for Distributed Tensor Resharding
Kaushik Kandadi, Youngeun Kwon, Sreeram Potluri +6
Distributed training and rollout generation often use different tensor layouts, requiring model weights to be resharded across distinct process groups. This M-to-N redistribution i…
Scalable Training of Mixture-of-Experts Models with Megatron Core
Zijie Yan, Hongxiao Bai, Xin Yao +42
Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total pa…
DeLTA: GPU Performance Model for Deep Learning Applications with In-depth Memory System Traffic Analysis
Sangkug Lym, Donghyuk Lee, Mike O'Connor +2
Training convolutional neural networks (CNNs) requires intense compute throughput and high memory bandwidth. Especially, convolution layers account for the majority of the executio…