LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

Yuhan Liu, Yihua Cheng, Jiayi Yao +8

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV cach…