2 papers
cs.DC2024
LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
Yi Xiong, Hao Wu, Changxu Shao +6
The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in ma…
cs.DC2024
vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
Jiale Xu, Rui Zhang, Cong Guo +11
Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. This surge in demand poses significant challenges in optimizing throughp…