REALM: Retrieval-Augmented Language Model Pre-Training

Abstract

Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Topics & keywords

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Bidirectional Attention Flow for Machine Comprehension
• End-To-End Memory Networks
• Semi-supervised Sequence Learning
• Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
• Learning Recurrent Span Representations for Extractive Question Answering
• Language Models as Knowledge Bases?
• A Retrieve-and-Edit Framework for Predicting Structured Outputs
• Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
• Large Memory Layers with Product Keys
• Generalization through Memorization: Nearest Neighbor Language Models
• A Discrete Hard EM Approach for Weakly Supervised Question Answering

Cited by in corpus (116)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Big Bird: Transformers for Longer Sequences
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Revisiting Few-sample BERT Fine-tuning
• Long Range Arena: A Benchmark for Efficient Transformers
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Pre-training via Paraphrasing
• Multimodal Few-Shot Learning with Frozen Language Models
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Language Models are Open Knowledge Graphs
• How Context Affects Language Models' Factual Predictions
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• Complementing Lexical Retrieval with Semantic Residual Embedding
• RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents
• Modifying Memories in Transformer Models
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Distilling Knowledge from Reader to Retriever for Question Answering
• Advanced Semantics for Commonsense Knowledge Extraction
• Generation-Augmented Retrieval for Open-domain Question Answering
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Example-Based Named Entity Recognition
• A Memory Efficient Baseline for Open Domain Question Answering
• Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
• Misinformation Has High Perplexity
• Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts
• Embedding-based Zero-shot Retrieval through Query Generation
• A Survey of Knowledge Enhanced Pre-trained Models
• Contextualized Representations Using Textual Encyclopedic Knowledge
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Nearest Neighbor Machine Translation
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
• End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training
• Language Models As or For Knowledge Bases
• Neural Machine Translation with Monolingual Translation Memory
• AmbigQA: Answering Ambiguous Open-domain Questions
• Pre-Trained Models: Past, Present and Future
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• The Life Cycle of Knowledge in Big Language Models: A Survey
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense
• Studying Strategically: Learning to Mask for Closed-book QA
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Optimizing Dense Retrieval Model Training with Hard Negatives
• Learning to Recombine and Resample Data for Compositional Generalization
• Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation
• Building Chinese Biomedical Language Models via Multi-Level Text Discrimination
• Accelerating Real-Time Question Answering via Question Generation
• More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering
• RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering
• Open-book Video Captioning with Retrieve-Copy-Generate Network
• Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
• Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
• DEEPAGÉ: Answering Questions in Portuguese about the Brazilian Environment
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
• Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
• Differentiable Retrieval Augmentation via Generative Language Modeling for E-commerce Query Intent Classification
• Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
• Human Language Modeling
• Learning Dense Representations of Phrases at Scale
• A Neighbourhood Framework for Resource-Lean Content Flagging
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
• Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering
• Taking Notes on the Fly Helps BERT Pre-training
• Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
• No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts
• Can NLI Models Verify QA Systems' Predictions?
• Visual Grounding Strategies for Text-Only Natural Language Processing
• PGT: Pseudo Relevance Feedback Using a Graph-Based Transformer
• Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
• TempEL: Linking Dynamically Evolving and Newly Emerging Entities
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Augmented Abstractive Summarization With Document-LevelSemantic Graph
• Adaptive Semiparametric Language Models
• Current Limitations of Language Models: What You Need is Retrieval
• Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
• Complementary Evidence Identification in Open-Domain Question Answering
• Towards Robust Neural Retrieval Models with Synthetic Pre-Training
• Simple Entity-Centric Questions Challenge Dense Retrievers
• Automatic Claim Review for Climate Science via Explanation Generation
• Frustratingly Hard Evidence Retrieval for QA Over Books
• Variational Learning for Unsupervised Knowledge Grounded Dialogs
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
• Zero-shot Slot Filling with DPR and RAG
• Syntax-Enhanced Pre-trained Model
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• Technical Question Answering across Tasks and Domains
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• Instance-Based Neural Dependency Parsing
• What makes us curious? analysis of a corpus of open-domain questions
• Unsupervised Open-Domain Question Answering
• Coarse-to-Fine Memory Matching for Joint Retrieval and Classification
• Bew: Towards Answering Business-Entity-Related Web Questions
• Large Product Key Memory for Pretrained Language Models
• Extreme Multi-label Learning for Semantic Matching in Product Search
• Efficient Retrieval Optimized Multi-task Learning
• Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks
• Generating multiple-choice questions for medical question answering with distractors and cue-masking
• Encapsulating Textual Contents into a MOC data Structure for Advanced Applications
• Relation-Guided Pre-Training for Open-Domain Question Answering
• Towards Universal Dense Retrieval for Open-domain Question Answering