BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

Yilong Zhao, Shuo Yang, Kan Zhu +5

Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applicat…