Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Yuyang Dai, Rana Shahout, Mahmood Sharif

Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize…