Quota-Based Aging for Prediction-Guided Scheduling in LLM Serving

Hanzhi Zhang, Anxuan Che, Hang Qi +2

Large language model (LLM) serving systems use token-level continuous batching to improve resource utilization, but heterogeneous response lengths complicate request scheduling. Pr…