CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing

Kuan Lu, Shuhang Lin, Sai Wu +7

Large language models (LLMs) are increasingly applied in long-context scenarios such as multi-turn conversations. However, long contexts pose significant challenges for inference e…