5 papers
OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang +2
Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LL…
Characterization-Guided GPU Fault Resilience in NVIDIA MPS
Rixin Liu, Xingqi Cui, Kaijian Wang +4
NVIDIA Multi-Process Service (MPS) enables fine-grained GPU sharing by allowing multiple processes to execute concurrently on the same GPU, making it an important mechanism for imp…
RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
Yifan Lu, Rixin Liu, Jiayi Yuan +4
Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become es…
From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models
Xingqi Cui, Chieh-Jan Mike Liang, Jiarong Xing +1
Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals…
Towards Efficient and Practical GPU Multitasking in the Era of LLM
Jiarong Xing, Yifan Qiao, Simon Mo +5
GPU singletasking is becoming increasingly inefficient and unsustainable as hardware capabilities grow and workloads diversify. We are now at an inflection point where GPUs must em…