5 papers
The Effect of Attention Head Count on Transformer Approximation
Penghao Yu, Haotian Jiang, Zeyu Bao +2
Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In…
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3
Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, w…
Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains
Peiran Zhou, Junnan Zhu, Yichen Shen +1
Large Language Models (LLMs) excel in language tasks but are prone to hallucinations and outdated knowledge. Retrieval-Augmented Generation (RAG) mitigates these by grounding LLMs…
Exploring Grokking: Experimental and Mechanistic Investigations
Hu Qiye, Zhou Hao, Yu RuoXi
The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero t…
How Transformers Get Rich: Approximation and Dynamics Analysis
Mingze Wang, Ruoxi Yu, Weinan E +1
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage e…