LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

Zhang Cao, Shujie Han, Juncheng Zhang +3

Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a wor…