Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Zihan Zhao, Baotong Lu, Shengjie Lin +8
Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent s…
cs.LG2024
DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
Yanqi Zhang, Yuwei Hu, Runyuan Zhao +2
Large language models (LLMs) demonstrate remarkable capabilities but face substantial serving costs due to their high memory demands, with the key-value (KV) cache being a primary…