15 citations · 27 across the 31 of their papers we have counts for
8 papers · 1 filter
SageBwd: A Trainable Low-bit Attention
Jintao Zhang, Marco Chen, Haoxu Wang +5
Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In pri…
SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training
Jintao Zhang, Haoxu Wang, Kai Jiang +6
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a…
Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization
Haocheng Xi, Shuo Yang, Yilong Zhao +13
Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregre…
FrontierCS: Evolving Challenges for Evolving Intelligence
Qiuyang Mang, Wenhao Chai, Zhifei Li +48
We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competiti…
vAttention: Verified Sparse Attention
Aditya Desai, Kumar Krishna Agrawal, Shuo Yang +5
State-of-the-art sparse attention methods for reducing decoding latency fall into two main categories: approximate top- (and its extension, top-) and recently introduced samp…
SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention
Jintao Zhang, Haoxu Wang, Kai Jiang +10
In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck due to the long sequence length and the quadratic complexity. We f…
S*: Test Time Scaling for Code Generation
Dacheng Li, Shiyi Cao, Chengkun Cao +6
Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the…
Twilight: Adaptive Attention Sparsity with Hierarchical Top- Pruning
Chaofan Lin, Jiaming Tang, Shuo Yang +6
Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-val…