Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving

Ke Cheng, Wen Hu, Zhi Wang +3

Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation lengt…