Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate

Fangyue Liu, Hua Liu, Xinyuan Lyu +8

LLM inference powers latency-critical production services nowadays. The bursty nature of inference traffic results in over-provisioning, which in turn leads to resource underutiliz…