ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor

Seungbeom Choi, Jeonghoe Goo, Eunjoo Jeon +2

We propose ELIS, a serving system for Large Language Models (LLMs) featuring an Iterative Shortest Remaining Time First (ISRTF) scheduler designed to efficiently manage inference t…