LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management

Yi Xiong, Hao Wu, Changxu Shao +6

The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in ma…