Past-Future Scheduler for LLM Serving under SLA Guarantees

Ruihao Gong, Shihao Bai, Siyu Wu +5

The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service fram…