NCCL M2N: A Layout- and Topology-Aware Collective for Distributed Tensor Resharding

Kaushik Kandadi, Youngeun Kwon, Sreeram Potluri +6

Distributed training and rollout generation often use different tensor layouts, requiring model weights to be resharded across distinct process groups. This M-to-N redistribution i…