Learning Dense Representations of Phrases at Scale

Abstract

Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

arXiv commentACL 2021. Code available at https://github.com/princeton-nlp/DensePhrases

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Skip-Thought Vectors
• REALM: Retrieval-Augmented Language Model Pre-Training
• End-to-End Open-Domain Question Answering with BERTserini
• Efficient Natural Language Response Suggestion for Smart Reply
• Learning Recurrent Span Representations for Extractive Question Answering
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering

Cited by in corpus (3)

• GNN-LM: Language Modeling based on Global Contexts via GNN
• Graph-free Multi-hop Reading Comprehension: A Select-to-Guide Strategy
• Zero-shot Slot Filling with DPR and RAG