Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference

Thomas Joshi, Herman Saini, Neil Dhillon +2

Large Language Models (LLMs) encounter severe memory inefficiencies during long-context inference due to conventional handling of key-value (KV) caches. In this work, we introduce…