4 papers
RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving
Wei Da, Evangelia Kalyvianaki
Heterogeneous LLM serving stacks split scheduling into two layers that optimize in isolation: model routers pick a model from quality and cost signals while ignoring instance load,…
LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
Wei Da, Evangelia Kalyvianaki
Realistic evaluation of LLM serving systems requires online workloads, dynamic arrivals, queueing, and the serving engine's local scheduling for execution batching, but running suc…
Dodoor: Efficient Randomized Decentralized Scheduling with Load Caching for Heterogeneous Tasks and Clusters
Wei Da, Evangelia Kalyvianaki
This paper presents Dodoor, a randomized decentralized scheduler for heterogeneous clusters. Dodoor removes hot-path probing via batched cache refreshes and introduces a heterogene…
Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling
Wei Da, Evangelia Kalyvianaki
This paper presents Astrolabe, a randomized prediction-guided scheduler for one-shot request dispatch in multi-instance large language model (LLM) serving. Astrolabe improves load…