Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs

Ferdi Kossmann, Bruce Fontaine, Daya Khudia +2

Serving systems for Large Language Models (LLMs) improve throughput by processing several requests concurrently. However, multiplexing hardware resources between concurrent request…