8 papers
Zepp: Accelerating Distributed MoE Serving under Relaxed Balance Constraints
Chang Chen, Andrew Yang, Tiancheng Chen +5
As Mixture-of-Experts (MoE) models continue to scale, serving them increasingly relies on expert parallelism (EP) across a growing number of devices. Yet skewed expert workloads cr…
AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking
Zaifeng Pan, Michael Wang, Chris Wu +4
LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimiza…
RR-Evict: Fine-Grained Prefix Cache Eviction beyond LRU for Agentic LLM Serving
Zaifeng Pan, Chris Wu, Zhengding Hu +3
LLM-based agents execute long-horizon tasks through repeated model calls interleaved with tool execution and user interaction. As each call extends the history accumulated in previ…
Argus: Orchestrating Cross-Layer GPU Performance Measurements around Semantic Regions
Jianzhu Yao, Yue Guan, Srivatsan Ramesh +11
GPU developers and automated optimizers need performance evidence for semantic code regions--such as neural-network operator implementations and pipeline stages--but this evidence…
Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting
Xinwei Qiang, Xiang Fang, Chang Chen +3
Block drafters propose several tokens in one forward pass, before earlier target tokens are realised. Their rejection mixes two losses: missing within-block path information and im…
GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving
Xinwei Qiang, Yifan Hu, Shixuan Sun +6
Diffusion Transformers (DiTs) have become the dominant architecture for image and video generation, creating growing demand for efficient DiT serving. Existing systems assign each…
DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
Xinwei Qiang, Hongmin Chen, Shixuan Sun +3
Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such…
Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
Xinwei Qiang, Yue Guan, Zhengding Hu +3
Communication has become a first-order bottleneck in large-scale GPU workloads, and existing distributed compilers address it mainly by overlapping whole compute and communication…