ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving

Yifan Qiao, Shu Anzai, Shan Yu +10

Large language model (LLM) serving demands low latency and high throughput, but high load variability makes it challenging to achieve high GPU utilization. In this paper, we identi…