19 citations · 19 across the 2 of their papers we have counts for
4 papers
Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
Chenyang Li, Yingyu Liang, Zhenmei Shi +2
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recen…
Fast Heavy Inner Product Identification Between Weights and Inputs in Neural Network Training
Lianke Qin, Saayan Mitra, Zhao Song +2
In this paper, we consider a heavy inner product identification problem, which generalizes the Light Bulb problem~(\cite{prr89}): Given two sets and $B \sub…
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Zichang Liu, Jue Wang, Tri Dao +8
Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference t…
Why Softmax Attention Outperforms Linear Attention
Yichuan Deng, Zhao Song, Kaijun Yuan +1
Large transformer models have achieved state-of-the-art results in numerous natural language processing tasks. Among the pivotal components of the transformer architecture, the att…