1 citations · 1 across the 2 of their papers we have counts for
2 papers
cs.LG2024
KV-weights are all you need for skipless transformers
Nils Graef
He and Hofmann (arXiv:2311.01906) detailed a skipless transformer without the V and P (post-attention projection) linear layers, which reduces the total number of weights. However,…
cs.LG2024★ 1 cited
Transformer tricks: Precomputing the first layer
Nils Graef
This micro-paper describes a trick to speed up inference of transformers with RoPE (such as LLaMA, Mistral, PaLM, and Gemma). For these models, a large portion of the first transfo…