10 papers
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Xiao Li, Chengruidong Zhang, Hao Luo +15
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active corr…
Region-Adaptive Sampling for Diffusion Transformers
Ziming Liu, Yifan Yang, Chengruidong Zhang +4
Diffusion models (DMs) have become the leading choice for generative tasks across diverse domains. However, their reliance on multiple sequential forward passes significantly limit…
MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
Wenxuan Li, Chengruidong Zhang, Huiqiang Jiang +3
The adoption of long context windows has become a standard feature in Large Language Models (LLMs), as extended contexts significantly enhance their capacity for complex reasoning…
Accelerating Prefilling via Decoding-time Contribution Sparsity
Zhiyuan He, Yike Zhang, Chengruidong Zhang +3
Large Language Models (LLMs) incur quadratic attention complexity with input length, creating a major time bottleneck in the prefilling stage. Existing acceleration methods largely…
SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
Yiqi Zhang, Huiqiang Jiang, Xufang Luo +7
Scaling reinforcement learning (RL) has shown strong promise for enhancing the reasoning abilities of large language models (LLMs), particularly in tasks requiring long chain-of-th…
LeanK: Learnable K Cache Channel Pruning for Efficient Decoding
Yike Zhang, Zhiyuan He, Huiqiang Jiang +4
Large language models (LLMs) enable long-context tasks but face efficiency challenges due to the growing key-value (KV) cache. We propose LeanK, a learning-based method that prunes…