RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving

Abstract

Current heterogeneous LLM serving clusters decouple request scheduling into two independent tiers: learned model routers select a model based on the prompt without visibility into instance-level load, while load balancers assign requests to an instance of that chosen model without cross-model quality or cost awareness. This decoupled hierarchy induces conflicting objectives and severely restricts the global optimization space. For example, a quality-oriented router may stubbornly direct 55% of requests to a 72B model regardless of growing queues, inflating mean latency by 2.4$\times$ under high load. Conversely, a cost-minimizing router over-allocates traffic to the cheapest model, leaving capable instances underutilized and capping aggregate cluster throughput. This paper presents RouteBalance, a serving scheduler that unifies model routing and instance dispatching into a single decision. RouteBalance dynamically batches incoming requests into short, load-adaptive rounds. In each round, it amortizes quality and response-length estimation across candidate models, takes a single snapshot of all instance loads, and solves a normalized assignment problem balancing quality, cost, and estimated latency. On a 28-GPU heterogeneous cluster spanning four Qwen model sizes and diverse GPU architectures, RouteBalance Pareto-dominates 231 of 246 baseline configurations across quality, latency, and cost simultaneously. These baselines comprise combinations of two learned routers with four downstream dispatchers, as well as five standalone dispatchers. At iso-quality, per-rate median reductions across baselines are 23--60% in mean E2E latency and 26--37% in serving cost. Code: https://github.com/AKafakA/route-balance.

arXiv comment17 pages including appendices, 5 figures. Substantially revised from v1: replaced the learned latency predictor with profiled TPOT estimates, added Gemma 3 and bursty-arrival experiments, and integrated appendices