2.4k citations · 3k across the 56 of their papers we have counts for
64 papers
Copy-Paste Language Models
Nathan Godey, Yoav Artzi
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit…
The Surprising Effectiveness of Shared Memory in Looped Transformers
Giovanni Monea, Keshav Ramji, Yousef El-Kurdi +4
Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cach…
Almost Free State Prediction Separation
John Langford, Nathan Godey, Giovanni Monea +5
State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forw…
Co-Evolving Structured Knowledge and Reasoning in Language Models
Ryan Thomas Noonan, Linxi Zhao, Menghan Xu +6
Retrieval-augmented methods improve factual accuracy by grounding language models in external knowledge, but retrieving over unstructured text often introduces irrelevant context a…
CT: Counterfactual Causal Reasoning for Sentiment Shifts in Social-Media Conversation Trees
S M Rafiuddin, Atriya Sen
Sentiment in social-media threads does not only vary across posts; it shifts as users react to claims, corrections, evidence, and hostility within a branching reply tree. We study…
Matryoshka Language Model Suites
Nathan Godey, Yoav Artzi
Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking su…