Service-Induced Congestion in Memory-Constrained LLM Serving

Ruicheng Ao, Jing Dong, Gan Luo +1

In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated to…