BatchGen: An Architecture for Scalable and Efficient Batch Inference

Tairan Xu, Leyang Xue, Zhan Lu +7

Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions…