15 papers
Understanding the Synchronization Tax in GPU Scale-Up Domains
Arjun Devraj, Lindsey Bowen, Rachee Singh
GPU scale-up domains have become the building block of modern machine learning infrastructure, and their design follows a clear trajectory of exponential growth in both interconnec…
CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure
Eric Ding, Byungsoo Oh, Bhaskar Kataria +8
Evaluative claims about LLM infrastructure -- ``workload X is fastest on hardware Y with software Z'' -- depend on a complex configuration space spanning hardware accelerators, int…
Eliminating Hidden Serialization in Multi-Node Megakernel Communication
Byungsoo Oh, Rachee Singh
Recent megakernel designs for Mixture-of-Experts (MoE) inference fuse expert computation with fine-grained, GPU-initiated communication into a single persistent GPU kernel, and out…
TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents
Abhishek Vijaya Kumar, Bhaskar Kataria, Byungsoo Oh +2
In RL post-training of LLM agents, calls to external tools take several seconds or even minutes, leaving allocated GPUs idle and inflating post-training time and cost. While many t…
Opus: Photonic Rail-Optimized Fabric in ML Datacenters
Eric Ding, Barry Lyu, Bhaskar Kataria +1
Rail-optimized network fabrics have become the de facto datacenter scale-out fabric for large-scale ML training. However, the use of high-radix electrical switches to provide all-t…
Stable and Fault-Tolerant Decentralized Traffic Engineering
Arjun Devraj, Umesh Krishnaswamy, Ying Zhang +4
Cloud providers have recently decentralized their wide-area network traffic engineering (TE) systems to contain the impact of TE controller failures. In the decentralized design, a…
PCCL: Photonic circuit-switched collective communication for distributed ML
Abhishek Vijaya Kumar, Arjun Devraj, Rachee Singh
Modern distributed ML suffers from a fundamental gap between the theoretical and realized performance of collective communication algorithms due to congestion and hop-count induced…
Morphlux: Transforming Torus Fabrics for Efficient Multi-tenant ML
Abhishek Vijaya Kumar, Eric Ding, Arjun Devraj +2
We develop Morphlux, a server-scale programmable photonic fabric to interconnect accelerators within servers. We show that augmenting state-of-the-art torus-based ML data-centers w…
Photonic Rails in ML Datacenters
Eric Ding, Chuhan Ouyang, Rachee Singh
Rail-optimized network fabrics have become the de facto datacenter scale-out fabric for large-scale ML training. However, the use of high-radix electrical switches to provide all-t…
FlashMoE: Fast Distributed MoE in a Single Kernel
Osayamen Jonathan Aimuyo, Byungsoo Oh, Rachee Singh
The computational sparsity of Mixture-of-Experts (MoE) models enables sub-linear growth in compute cost as model size increases, thus offering a scalable path to training massive n…
Efficient AllReduce with Stragglers
Arjun Devraj, Eric Ding, Abhishek Vijaya Kumar +2
Distributed machine learning workloads use data and tensor parallelism for training and inference, both of which rely on the AllReduce collective to synchronize gradients or activa…
LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabrics
Abhishek Vijaya Kumar, Eric Ding, Arjun Devraj +2
When accelerators fail in modern ML datacenters, operators migrate the affected ML training or inference jobs to entirely new racks. This approach, while preserving network perform…
Chip-to-chip photonic connectivity in multi-accelerator servers for ML
Abhishek Vijaya Kumar, Arjun Devraj, Darius Bunandar +1
We present a rack-scale compute architecture for ML using multi-accelerator servers connected via chip-to-chip silicon photonic components. Our architecture achieves (1) multi-tena…
PipSwitch: A Circuit Switch Using Programmable Integrated Photonics
Eric Ding, Rachee Singh
We present an optical circuit switch design for programmable integrated photonics (PIPs). Our solution finds the correct and optimal set of matchings that provides all-to-all netwo…
AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
Abhishek Vijaya Kumar, Gianni Antichi, Rachee Singh
Inference on large-language models (LLMs) is constrained by GPU memory capacity. A sudden increase in the number of inference requests to a cloud-hosted LLM can deplete GPU memory,…