Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models

Junxiang Qiu, Shuo Wang, Zhengsu Chen +4

Attention serves as the fundamental mechanism for long-context modeling in large language models (LLMs), yet dense attention becomes structurally prohibitive for long sequences due…