9 citations · 14 across the 7 of their papers we have counts for
7 papers
Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
Franz A. Heinsen, Leo Kozachkov
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attention incurs costs that increase with conte…
Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation
Franz A. Heinsen, Leo Kozachkov
Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce gener…
Softmax Attention with Constant Cost per Token
Franz A. Heinsen
We propose a simple modification to the conventional attention mechanism applied by Transformers: Instead of quantifying pairwise query-key similarity with scaled dot-products, we…
Efficient Parallelization of a Ubiquitous Sequential Computation
Franz A. Heinsen
We find a succinct expression for computing the sequence in parallel with two prefix sums, given , , $b_t \in…
An Algorithm for Routing Vectors in Sequences
Franz A. Heinsen
We propose a routing algorithm that takes a sequence of vectors and computes a new sequence with specified length and vector size. Each output vector maximizes "bang per bit," the…
Tree Methods for Hierarchical Classification in Parallel
Franz A. Heinsen
We propose methods that enable efficient hierarchical classification in parallel. Our methods transform a batch of classification scores and labels, corresponding to given nodes in…