4 citations · 5 across the 4 of their papers we have counts for
5 papers
Kwai Summary Attention V2 Technical Report
OneRec Team, Chenglong Chu, Guohong Mu +30
Summary-token architectures such as Kwai Summary Attention (KSA) compress distant context into learnable Summary token per chunk, and hybrid variants interleave these layers with f…
Kwai Summary Attention Technical Report
Chenglong Chu, Guorui Zhou, Guowang Zhang +35
Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agen…
Retrieval Head Mechanistically Explains Long-Context Factuality
Wenhao Wu, Yizhong Wang, Guangxuan Xiao +2
Despite the recent progress in long-context language models, it remains elusive how transformer-based models exhibit the capability to retrieve relevant information from arbitrary…
Data Engineering for Scaling Language Models to 128K Context
Yao Fu, Rameswar Panda, Xinyao Niu +4
We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in part…
FiLM: Fill-in Language Models for Any-Order Generation
Tianxiao Shen, Hao Peng, Ruoqi Shen +3
Language models have become the backbone of today's AI systems. However, their predominant left-to-right generation limits the use of bidirectional context, which is essential for…