4 papers
PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
Hong Liu, Yuan Cheng, Lin Niu +5
Token-level sparse attention, as implemented by DeepSeek Sparse Attention (DSA) in production systems, makes the downstream attention efficient but shifts the bottleneck to the ind…
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
Junhao Hu, Fangze Li, Mingtao Xu +9
Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference e…
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
Anmin Liu, Ruixuan Yang, Huiqiang Jiang +5
Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. Whi…
KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Bandits
Dezhi Ran, Shuxiao Xie, Mingfang Ji +9
High-performance GPU kernels are critical for efficient LLM serving, yet their optimization remains a bottleneck requiring deep system expertise. While code LLMs show promise in ge…