OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving

Siyu Wu, Zihan Tang, Yuting Zeng +5

Large Language Models (LLMs) are increasingly deployed in both latency-sensitive online services and cost-sensitive offline workloads. Co-locating these workloads on shared serving…