Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays

Tianrui Liu, Xiaohai Hu

Large language model (LLM) inference deployed on Kubernetes faces an autoscaling challenge that conventional web services do not: new serving replicas take two to ten minutes to st…