SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

Yinghao Tang, Tingfeng Lan, Xiuqi Huang +2

Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Pe…