4 papers
LUMEN: Coordinated Failure Recovery for Distributed LLM Serving
Zhang Cao, Shujie Han, Juncheng Zhang +3
Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a wor…
Leveraging I/O Stalls for Efficient Scheduling in ANNS
Juncheng Zhang, Yuanming Ren, Yongkun Li +1
Disk-based graph indexes for approximate nearest neighbor search (ANNS) must serve latency-sensitive queries and throughput-demanding updates concurrently. We observe that over 40%…
TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
Shujie Han, Feng Jiang, Patrick P. C. Lee +5
Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkp…
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Qingxiu Liu, Yongchao He, Runhan Jiang +4
Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inferenc…