From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models

Xingqi Cui, Chieh-Jan Mike Liang, Jiarong Xing +1

Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals…