OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang +2

Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LL…