Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

Ashman Mehra, Akshan Agrawal, Akshan Agarwal +1

Large language model serving is increasingly limited not only by model size and GPU speed, but also by scheduling during autoregressive decoding, where continuous batching interlea…