3 papers
cs.LG2026
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
Qiuwu Chen, Zimo Liu, Yuchen Li +8
Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limi…
cs.CL2026
Training-free Context-adaptive Attention for Efficient Long Context Modeling
Zeng You, Yaofo Chen, Shuhai Zhang +5
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. These capabilities stem primarily from the self-att…
cs.CL2025
Curse of High Dimensionality Issue in Transformer for Long-context Modeling
Shuhai Zhang, Zeng You, Yaofo Chen +5
Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-cont…