Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

Shaoke Fang, Ziang Li, Wenfei Wu +3

Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive…