DLB: Distributed Load Balancing at Scale for Generative AI Inference

Santiago R. Balseiro, Bartek Wydrowski, Sameer Agarwal +11

The reliance on scarce and expensive accelerators such as GPUs and TPUs in modern datacenters places unprecedented demands on backend infrastructure. For workloads characterized by…