OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

Can Xiao, Sukmin Cho, Junbong We +7

Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the…