1 citations · 1 across the 13 of their papers we have counts for
4 papers · 2 filters
Copy-Paste Language Models
Nathan Godey, Yoav Artzi
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit…
Co-LMLM: Continuous-Query Limited Memory Language Models
Yair Feldman, Linxi Zhao, Nathan Godey +5
Externalizing knowledge in LLM pre-training is a promising avenue to achieve higher performance at smaller scales, control knowledge use, and overall increase model transparency. W…
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley +1
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction sep…
Lost in Backpropagation: The LM Head is a Gradient Bottleneck
Nathan Godey, Yoav Artzi
The last layer of neural language models (LMs) projects output features of dimension to logits in dimension , the size of the vocabulary, where usually . This misma…