Adaptive Inference Batching using Policy Gradients

Ruslan Sharifullin

Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual…