LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

Haocheng Xia, Mihir Pamnani, Hanxi Fang +2

Key-value (KV) caching accelerates inference of large language models (LLMs) by reusing past computations for generated tokens. Its importance becomes even greater in long-context…