Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Siyu Song, Qi Bai, Jinbo Hao +3

Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked…