3 papers
cs.CL2026
Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
Jiaqi Leng, Xiang Hu, Junxiong Wang +3
Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alt…
cs.CL2026
Distilling Token-Trained Models into Byte-Level Models
Zishuo Bao, Jiaqi Leng, Junxiong Wang +2
Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on…
cs.CL2025
Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
Xiang Hu, Jiaqi Leng, Jun Zhao +2
A key advantage of Recurrent Neural Networks (RNNs) over Transformers is their linear computational and space complexity enables faster training and inference for long sequences. H…