45 citations · 157 across the 16 of their papers we have counts for
16 papers
Integrating Planning into Single-Turn Long-Form Text Generation
Yi Liang, You Wu, Honglei Zhuang +8
Generating high-quality, in-depth textual documents, such as academic papers, news articles, Wikipedia entries, and books, remains a significant challenge for Large Language Models…
Functional Interpolation for Relative Positions Improves Long Context Transformers
Shanda Li, Chong You, Guru Guruganesh +7
Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Thou…
MEMORY-VQ: Compression for Tractable Internet-Scale Memory
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis +4
Retrieval augmentation is a powerful but expensive method to make language models more knowledgeable about the world. Memory-based methods like LUMEN pre-compute token representati…
GLIMMER: generalized late-interaction memory reranker
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +3
Memory-augmentation is a powerful approach for efficiently incorporating external information into language models, but leads to reduced performance relative to retrieving text. Re…
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong +3
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not…
CoLT5: Faster Long-Range Transformers with Conditional Computation
Joshua Ainslie, Tao Lei, Michiel de Jong +9
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but…
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald +4
Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks. However, they are also expensive…
ImPaKT: A Dataset for Open-Schema Knowledge Base Construction
Luke Vilnis, Zach Fisher, Bhargav Kanagal +2
Large language models have ushered in a golden age of semantic parsing. The seq2seq paradigm allows for open-schema and abstractive attribute and relation extraction given only sma…
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie +4
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the architecture used for Fi…
Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models
Luke Vilnis, Yury Zemlyanskiy, Patrick Murray +2
Decoding methods for large language models often trade-off between diversity of outputs and parallelism of computation. Methods such as beam search and Gumbel top-k sampling can gu…
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie +5
A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this…
MAVE: A Product Dataset for Multi-source Attribute Value Extraction
Li Yang, Qifan Wang, Zac Yu +5
Attribute value extraction refers to the task of identifying values of an attribute of interest from product information. Product attribute values are essential in many e-commerce…
ShopTalk: A System for Conversational Faceted Search
Gurmeet Manku, James Lee-Thorp, Bhargav Kanagal +11
We present ShopTalk, a multi-turn conversational faceted search system for shopping that is designed to handle large and complex schemas that are beyond the scope of state of the a…
ETC: Encoding Long and Structured Inputs in Transformers
Joshua Ainslie, Santiago Ontanon, Chris Alberti +7
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transforme…
An Empirical Comparison of Algorithms for Aggregating Expert Predictions
Varsha Dani, Omid Madani, David M Pennock +2
Predicting the outcomes of future events is a challenging problem for which a variety of solution methods have been explored and attempted. We present an empirical comparison of a…
Relational Dynamic Bayesian Networks
P. Domingos, S. Sanghai, D. Weld
Stochastic processes that involve the creation of objects and relations over time are widespread, but relatively poorly studied. For example, accurate fault diagnosis in factory as…