Stable Token-level Scheduling for Serverless LLM Inference

Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti Addya

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization throug…