RAC: Relation-Aware Cache Replacement for Large Language Models

Yuchong Wu, Zihuan Xu, Wangze Ni +5

The scaling of Large Language Model (LLM) services faces significant cost and latency challenges, making effective caching under tight capacity crucial. Existing cache replacement…