Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache

Hanchen Li, Yuhan Liu, Yihua Cheng +2

Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM in…