4 papers
The Effect of Attention Head Count on Transformer Approximation
Penghao Yu, Haotian Jiang, Zeyu Bao +2
Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In…
Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains
Peiran Zhou, Junnan Zhu, Yichen Shen +1
Large Language Models (LLMs) excel in language tasks but are prone to hallucinations and outdated knowledge. Retrieval-Augmented Generation (RAG) mitigates these by grounding LLMs…
Exploring Grokking: Experimental and Mechanistic Investigations
Hu Qiye, Zhou Hao, Yu RuoXi
The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero t…
How Transformers Get Rich: Approximation and Dynamics Analysis
Mingze Wang, Ruoxi Yu, Weinan E +1
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage e…