Evaluating the accuracy of KV cache reuse techniques

Samuel Cestola, Tianxiang Xia, Pengfei Zheng +4

Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV…