Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service

Xianzhe Zheng, Zhengheng Wang, Ruiyan Ma +17

The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity m…