Publications (38)
Command A: An Enterprise-Ready Large Language Model
Team Cohere, :, Aakanksha +227
In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command A is an agent-optimised…
Autoregressive Search Engines: Generating Substrings as Document Identifiers
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis +3
Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus. Autoregressive language models…
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
Patrick Lewis, Yuxiang Wu, Linqing Liu +5
Open-domain Question Answering models which directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of spee…
From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models
Luiza Pozzobon, Patrick Lewis, Sara Hooker +1
To date, toxicity mitigation in language models has almost entirely been focused on single-language settings. As language models embrace multilingual capabilities, it's crucial our…
Interpretation of Natural Language Rules in Conversational Machine Reading
Marzieh Saeidi, Max Bartolo, Patrick Lewis +5
Most work in machine reading focuses on question answering problems where the answer is directly expressed in the text to read. However, many real-world question answering problems…
MLQA: Evaluating Cross-lingual Extractive Question Answering
Patrick Lewis, Barlas OÄuz, Ruty Rinott +2
Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to…
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin +8
In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-doma…
Reasoning over Public and Private Data in Retrieval-Based Systems
Simran Arora, Patrick Lewis, Angela Fan +2
Users and organizations are generating ever-increasing amounts of private data from a wide range of sources. Incorporating private data is important to personalize open-domain appl…
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models
Luiza Pozzobon, Beyza Ermis, Patrick Lewis +1
Considerable effort has been dedicated to mitigating toxicity, but existing methods often require drastic modifications to model parameters or the use of computationally intensive…
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training
Kelly Marchisio, Patrick Lewis, Yihong Chen +1
Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body fro…
EditEval: An Instruction-Based Benchmark for Text Improvements
Jane Dwivedi-Yu, Timo Schick, Zhengbao Jiang +6
Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text. Writing, however, is naturally an iterativ…
KILT: a Benchmark for Knowledge Intensive Language Tasks
Fabio Petroni, Aleksandra Piktus, Angela Fan +10
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models d…
Dense Passage Retrieval for Open-Domain Question Answering
Vladimir Karpukhin, Barlas OÄuz, Sewon Min +5
Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de…
MultiContrievers: Analysis of Dense Retrieval Representations
Seraphina Goldfarb-Tarrant, Pedro Rodriguez, Jane Dwivedi-Yu +1
Dense retrievers compress source documents into (possibly lossy) vector representations, yet there is little analysis of what information is lost versus preserved, and how it affec…
Challenges in Generalization in Open Domain Question Answering
Linqing Liu, Patrick Lewis, Sebastian Riedel +1
Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with trai…
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer +8
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datas…
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Gautier Izacard, Patrick Lewis, Maria Lomeli +7
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question an…
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research
Luiza Pozzobon, Beyza Ermis, Patrick Lewis +1
Perception of toxicity evolves over time and often differs between geographies and cultural backgrounds. Similarly, black-box commercially available APIs for detecting toxicity, su…
Improving Wikipedia Verifiability with AI
Fabio Petroni, Samuel Broscheit, Aleksandra Piktus +10
Verifiability is a core content policy of Wikipedia: claims that are likely to be challenged need to be backed by citations. There are millions of articles available online and tho…
Boosted Dense Retriever
Patrick Lewis, Barlas OÄuz, Wenhan Xiong +3
We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on…
Domain-matched Pre-training Tasks for Dense Retrieval
Barlas OÄuz, Kushal Lakhotia, Anchit Gupta +8
Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks. A notable exception is information ret…
FLARE: Faithful Logic-Aided Reasoning and Exploration
Erik Arakelyan, Pasquale Minervini, Pat Verga +2
Modern Question Answering (QA) and Reasoning approaches based on Large Language Models (LLMs) commonly use prompting techniques, such as Chain-of-Thought (CoT), assuming the result…
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus +9
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. Howe…
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Xilun Chen, Kushal Lakhotia, Barlas OÄuz +6
Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare…
A Few More Examples May Be Worth Billions of Parameters
Yuval Kirstain, Patrick Lewis, Sebastian Riedel +1
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while sc…
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
Sewon Min, Jordan Boyd-Graber, Chris Alberti +50
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and…
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Patrick Lewis, Pontus Stenetorp, Sebastian Riedel
Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question f…
ALR: A Retrieve-then-Reason Framework for Long-context Question Answering
Huayang Li, Pat Verga, Priyanka Sen +5
The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capabi…
Unsupervised Question Decomposition for Question Answering
Ethan Perez, Patrick Lewis, Wen-tau Yih +2
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling questions with…
PEER: A Collaborative Language Model
Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang +7
Textual content is often the output of a collaborative writing process: We start with an initial draft, ask for suggestions, and repeatedly make changes. Agnostic of this process,…
Unsupervised Question Answering by Cloze Translation
Patrick Lewis, Ludovic Denoyer, Sebastian Riedel
Obtaining training data for Question Answering (QA) is time-consuming and resource-intensive, and existing QA datasets are only available for limited domains and languages. In this…
How Context Affects Language Models' Factual Predictions
Fabio Petroni, Patrick Lewis, Aleksandra Piktus +4
When pre-trained on large unsupervised textual corpora, language models are able to store and retrieve factual knowledge to some extent, making it possible to use them directly for…
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
Pat Verga, Sebastian Hofstatter, Sophia Althammer +6
As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe parti…
SnapKV: LLM Knows What You are Looking for Before Generation
Yuhong Li, Yingbing Huang, Bowen Yang +6
Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. Howe…
Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models
Xinyu Zhang, Sebastian Hofstätter, Patrick Lewis +2
Listwise rerankers based on large language models (LLM) are the zero-shot state-of-the-art. However, current works in this direction all depend on the GPT models, making it a singl…
Counsel: A Meta-Evaluation Dataset for Agentic Tasks
Sashank Pisupati, Henry Broomfield, Eujeong Choi +5
As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agen…
Task-aware Retrieval with Instructions
Akari Asai, Timo Schick, Patrick Lewis +5
We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries. We aim to develop a general-purpos…
Language Models as Knowledge Bases?
Fabio Petroni, Tim Rocktäschel, Patrick Lewis +4
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models…