3 papers
cs.CL2026
Why Softmax Attention Outperforms Linear Attention
Yichuan Deng, Zhao Song, Kaijun Yuan +1
Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the att…
cs.DS2025
Faster Algorithm for Structured John Ellipsoid Computation
Yang Cao, Xiaoyu Li, Zhao Song +2
The famous theorem of Fritz John states that any convex body has a unique maximal volume inscribed ellipsoid, known as the John Ellipsoid. Computing the John Ellipsoid is a fundame…
cs.LG2025
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
Chenyang Li, Yingyu Liang, Zhenmei Shi +2
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recen…