Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

Li Kong, Qi Qi, Yinyu Ye +1

The rapid growth of interactive Large Language Model serving has made efficient management of dynamic Key-Value cache footprints increasingly important for inference performance. M…