Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

Li Kong, Qi Qi, Yinyu Ye +1

The explosive demand for interactive Large Language Model serving has highlighted the management of the Key-Value cache's dynamic memory footprint as a critical area for performanc…