2 papers
cs.CL2026
The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends
Zhentao Tan, Jingyi Shen, Yanbo Li +3
Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context len…
cs.CL2026
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Zhentao Tan, Wei Chen, Jingyi Shen +4
The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a…