2 papers
cs.DC2026
NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
Amos Goldman, Nimrod Boker, Maayan Sheraizin +15
Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such…
cs.LG2020
O(1) Communication for Distributed SGD through Two-Level Gradient Averaging
Subhadeep Bhattacharya, Weikuan Yu, Fahim Tahmid Chowdhury
Large neural network models present a hefty communication challenge to distributed Stochastic Gradient Descent (SGD), with a communication complexity of O(n) per worker for a model…