Showing 2025Show all
3 papers · 1 filter
cs.LG2025
Understanding Transformer from the Perspective of Associative Memory
Shu Zhong, Mingyu Xu, Tenglong Ao +1
In this paper, we share our reflections and insights on understanding Transformer architectures through the lens of associative memory--a classic psychological concept inspired by…
cs.CL2025
Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
Yao Xu, Mingyu Xu, Fangyu Lei +7
Recently, models such as OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable performance on complex reasoning tasks through Long Chain-of-Thought (Long-CoT) reasoning. Although…
cs.CL2025
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
Zican Dong, Junyi Li, Jinhao Jiang +4
Large language models (LLMs) have gained extended context windows through scaling positional encodings and lightweight continual pre-training. However, this often leads to degraded…