HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location

Ting Sun, Penghan Wang, Fan Lai

Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatb…