15 papers
Matryoshka Language Model Suites
Nathan Godey, Yoav Artzi
Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking su…
Lost in Backpropagation: The LM Head is a Gradient Bottleneck
Nathan Godey, Yoav Artzi
The last layer of neural language models (LMs) projects output features of dimension to logits in dimension , the size of the vocabulary, where usually . This misma…
Co-LMLM: Continuous-Query Limited Memory Language Models
Yair Feldman, Linxi Zhao, Nathan Godey +5
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the…
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley +1
Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \emph{state-prediction sep…
Post-training for Efficient Communication via Convention Formation
Yilun Hua, Evan Wang, Yoav Artzi
Humans communicate with increasing efficiency in multi-turn interactions, by adapting their language and forming ad-hoc conventions. In contrast, prior work shows that LLMs do not…
No Mean Feat: Simple, Strong Baselines for Context Compression
Yair Feldman, Yoav Artzi
Context compression reduces Transformer inference costs by replacing lengthy inputs with shorter pre-computed representations. It carries significant benefits for retrieval-augment…