8 citations · 13 across the 4 of their papers we have counts for
6 papers · 1 filter
MEMORY-VQ: Compression for Tractable Internet-Scale Memory
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis +4
Retrieval augmentation is a powerful but expensive method to make language models more knowledgeable about the world. Memory-based methods like LUMEN pre-compute token representati…
GLIMMER: generalized late-interaction memory reranker
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +3
Memory-augmentation is a powerful approach for efficiently incorporating external information into language models, but leads to reduced performance relative to retrieving text. Re…
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong +3
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not…
CoLT5: Faster Long-Range Transformers with Conditional Computation
Joshua Ainslie, Tao Lei, Michiel de Jong +9
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but…
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie +5
A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this…
ReadTwice: Reading Very Large Documents with Memories
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong +3
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose…