Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective

Fangzhou Wu, Sandeep Silwal, Qiuyi +1

KV caching is a fundamental technique for accelerating Large Language Model (LLM) inference by reusing key-value (KV) pairs from previous queries, but its effectiveness under limit…