Online Batching with Lookahead: Adversarial Limits and Stochastic Gains

Dimitris Christou, Jiayi Chen, Nihal Sharma +3

Modern inference systems rely on batching to exploit parallel execution and amortize serial execution overheads, but batching creates a fundamental online scheduling tradeoff: serv…