Efficient LLM Scheduling by Learning to Rank

Yichao Fu, Siqi Zhu, Runlong Su +3

In Large Language Model (LLM) inference, the output length of an LLM request is typically regarded as not known a priori. Consequently, most LLM serving systems employ a simple Fir…