4 papers · 1 filter
The Effect of Attention Head Count on Transformer Approximation
Penghao Yu, Haotian Jiang, Zeyu Bao +2
Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In…
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3
Transformers have achieved remarkable successes across a wide range of applications, yet the theoretical foundation of their model efficiency remains underexplored. In this work, w…
Exploring Grokking: Experimental and Mechanistic Investigations
Hu Qiye, Zhou Hao, Yu RuoXi
The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero t…
How Transformers Get Rich: Approximation and Dynamics Analysis
Mingze Wang, Ruoxi Yu, Weinan E +1
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage e…