Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

Yiheng Tao, Yihe Zhang, Matthew Dearing +4

Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with t…