An experimental study of KV cache reuse strategies in chunk-level caching systems

Samuel Cestola, Tianxiang Xia, Zheng Weiyan +2

Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precompu…