HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

Xingyu Qu, Tianhao Lin, Yiqi Li +2

Online large language model (LLM) services process requests that share context but differ in prefix structure and arrival timing. Finite capacity couples request admission and key-…