1 citations · 2 across the 4 of their papers we have counts for
4 papers
Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
Josh Alman, Zhao Song
Attention mechanisms lie at the heart of modern large language models (LLMs). Straightforward algorithms for forward and backward (gradient) computation take quadratic time, and a…
Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform
Josh Alman, Zhao Song
The transformer architecture has been widely applied to many machine learning tasks. A main bottleneck in the time to perform transformer computations is a task called attention co…
The Fine-Grained Complexity of Gradient Computation for Training Large Language Models
Josh Alman, Zhao Song
Large language models (LLMs) have made fundamental contributions over the last a few years. To train an LLM, one needs to alternatingly run `forward' computations and `backward' co…
How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation
Josh Alman, Zhao Song
In the classical transformer attention scheme, we are given three size matrices (the query, key, and value tokens), and the goal is to compute a new $n \time…