Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

Mert Yildiz, Pietro Spadaccino, Alexey Rolich +2

Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardw…