29 citations · 60 across the 14 of their papers we have counts for
14 papers
Composer 2 Technical Report
Cursor Research, :, Aaron Chan +53
Composer 2 is a specialized model designed for agentic software engineering. The model demonstrates strong long-term planning and coding intelligence while maintaining the ability…
MEMORY-VQ: Compression for Tractable Internet-Scale Memory
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis +4
Retrieval augmentation is a powerful but expensive method to make language models more knowledgeable about the world. Memory-based methods like LUMEN pre-compute token representati…
GLIMMER: generalized late-interaction memory reranker
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +3
Memory-augmentation is a powerful approach for efficiently incorporating external information into language models, but leads to reduced performance relative to retrieving text. Re…
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong +3
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not…
CoLT5: Faster Long-Range Transformers with Conditional Computation
Joshua Ainslie, Tao Lei, Michiel de Jong +9
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but…
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +4
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks. However, they are also expensive…
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie +4
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the architecture used for Fi…
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie +5
A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this…
QA Is the New KR: Question-Answer Pairs as Knowledge Bases
Wenhu Chen, William W. Cohen, Michiel De Jong +4
In this position paper, we propose a new approach to generating a type of knowledge base (KB) from text, based on question generation and entity linking. We argue that the proposed…
Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
Wenhu Chen, Pat Verga, Michiel de Jong +2
Retrieval augmented language models have recently become the standard for knowledge intensive tasks. Rather than relying purely on latent semantics within the parameters of large n…
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +2
Natural language understanding tasks such as open-domain question answering often require retrieving and assimilating factual information from multiple sources. We propose to addre…
ReadTwice: Reading Very Large Documents with Memories
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong +3
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose…
Neural Theorem Provers Do Not Learn Rules Without Exploration
Michiel de Jong, Fei Sha
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks. The Neural Theorem Proving (NTP) model by Rockt…
Weighted Global Normalization for Multiple Choice Reading Comprehension over Long Documents
Aditi Chaudhary, Bhargavi Paranjape, Michiel de Jong
Motivated by recent evidence pointing out the fragility of high-performing span prediction models, we direct our attention to multiple choice reading comprehension. In particular,…