2 papers
cs.AI2025
Using Span Queries to Optimize for Cache and Attention Locality
Paul Castro, Nick Mitchell, Nathan Ordonez +3
Clients are evolving beyond chat completion, and now include a variety of innovative inference-time scaling and deep reasoning techniques. At the same time, inference servers remai…
cs.LG2025
Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
Thomas Joshi, Herman Saini, Neil Dhillon +2
Large Language Models (LLMs) encounter severe memory inefficiencies during long-context inference due to conventional handling of key-value (KV) caches. In this work, we introduce…