3 citations · 3 across the 19 of their papers we have counts for
6 papers · 1 filter
Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Zhigeng Liu, Zhiyuan Ning, Ruixiao Li +5
The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding.…
Prism: Spectral-Aware Block-Sparse Attention
Xinghao Wang, Pengyu Wang, Xiaoran Liu +4
Block-sparse attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically empl…
AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
Shicheng Fang, Yuxin Wang, Xiaoran Liu +6
The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static,…
MrRoPE: Mixed-radix Rotary Position Embedding
Qingyuan Tian, Wenhong Zhu, Xiaoran Liu +2
Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-tra…
DiRL: An Efficient Post-Training Framework for Diffusion Language Models
Ying Zhu, Jiaxin Wan, Xiaoran Liu +7
Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models. While recent efforts have validated their pre-training potential and accele…
FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
Siyang He, Qiqi Wang, Xiaoran Liu +8
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of ar…