Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Abstract

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

Accepted at NeurIPS 2020

References in corpus (18)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Mixed Precision Training
• End-To-End Memory Networks
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• REALM: Retrieval-Augmented Language Model Pre-Training
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Passage Re-ranking with BERT
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Release Strategies and the Social Impacts of Language Models
• When Will AI Exceed Human Performance? Evidence from AI Experts
• Hierarchical Neural Story Generation
• A Retrieve-and-Edit Framework for Predicting Structured Outputs
• How Context Affects Language Models' Factual Predictions
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Large Memory Layers with Product Keys
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• Reasoning Over Semantic-Level Graph for Fact Checking

Cited by in corpus (273)

• Language Models are Few-Shot Learners
• Large language models in medicine: the potentials and pitfalls
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Big Bird: Transformers for Longer Sequences
• A Survey of Knowledge-Enhanced Text Generation
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• The Power of Noise: Redefining Retrieval for RAG Systems
• Autoregressive Entity Retrieval
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• NetLLM: Adapting Large Language Models for Networking
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering
• Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering
• Evaluation of Retrieval-Augmented Generation: A Survey
• Delving into LLM-assisted writing in biomedical publications through excess vocabulary
• Large language models surpass human experts in predicting neuroscience results
• ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Framework
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
• How understanding large language models can inform the use of ChatGPT in physics education
• Vector database management systems: Fundamental concepts, use-cases, and current challenges
• Generative Artificial Intelligence in Learning Analytics: Contextualising Opportunities and Challenges through the Learning Analytics Cycle
• Woodpecker: Hallucination Correction for Multimodal Large Language Models
• Advancing Building Energy Modeling with Large Language Models: Exploration and Case Studies
• Pre-training via Paraphrasing
• Multimodal Few-Shot Learning with Frozen Language Models
• SelfKG: Self-Supervised Entity Alignment in Knowledge Graphs
• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?
• Ontology-enhanced Prompt-tuning for Few-shot Learning
• GERE: Generative Evidence Retrieval for Fact Verification
• A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods
• Retrieving Supporting Evidence for Generative Question Answering
• CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
• Leveraging LLMs for the Quality Assurance of Software Requirements
• Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods
• Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)
• Enhancing Knowledge Retrieval with In-Context Learning and Semantic Search through Generative AI
• MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
• Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy
• Fact-checking information from large language models can decrease headline discernment
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
• Application of NotebookLM, a Large Language Model with Retrieval-Augmented Generation, for Lung Cancer Staging
• From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents
• Knowledge-Aware Language Model Pretraining
• OpenFOAMGPT: a RAG-Augmented LLM Agent for OpenFOAM-Based Computational Fluid Dynamics
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• Modifying Memories in Transformer Models
• Quo Vadis ChatGPT? From Large Language Models to Large Knowledge Models
• LLM-based agents for automating the enhancement of user story quality: An early report
• Enhancing textual textbook question answering with large language models and retrieval augmented generation
• Distilling Knowledge from Reader to Retriever for Question Answering
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis
• Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insights
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
• LessonPlanner: Assisting Novice Teachers to Prepare Pedagogy-Driven Lesson Plans with Large Language Models
• An Evaluation on Large Language Model Outputs: Discourse and Memorization
• Generation-Augmented Retrieval for Open-domain Question Answering
• Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study
• Automated Review Generation Method Based on Large Language Models
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
• KNOWNET: Guided Health Information Seeking from LLMs via Knowledge Graph Integration
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Controllable Semantic Parsing via Retrieval Augmentation
• A Memory Efficient Baseline for Open Domain Question Answering
• PAM: Understanding Product Images in Cross Product Category Attribute Extraction
• Fine-tuning a Large Language Model for Automating Computational Fluid Dynamics Simulations
• Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detection
• Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
• Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
• Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency
• Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
• Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education
• Supporting Energy Policy Research with Large Language Models
• Is Retriever Merely an Approximator of Reader?
• On the Evaluation of Machine-Generated Reports
• Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts
• EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation
• Intelligent Computing Social Modeling and Methodological Innovations in Political Science in the Era of Large Language Models
• Learning to Rewrite Prompts for Personalized Text Generation
• RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements
• Neural Language Generation: Formulation, Methods, and Evaluation
• StoryVerse: Towards Co-authoring Dynamic Plot with LLM-based Character Simulation via Narrative Planning
• Exploring Gen-AI applications in building research and industry: A review
• Exploring Multilingual Large Language Models for Enhanced TNM classification of Radiology Report in lung cancer staging
• Multimodal Large Language Models for Bioimage Analysis
• EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
• Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
• A Reliable Knowledge Processing Framework for Combustion Science using Foundation Models
• To Search or To Gen? Exploring the Synergy between Generative AI and Web Search in Programming
• Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
• Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
• Opportunities and Challenges of Generative-AI in Finance
• Nearest Neighbor Machine Translation
• Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graph
• Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
• Multi-Meta-RAG: Improving RAG for Multi-Hop Queries using Database Filtering with LLM-Extracted Metadata
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach
• Cross-Domain Aspect Extraction using Transformers Augmented with Knowledge Graphs
• Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies
• Alleviating the Long-Tail Problem in Conversational Recommender Systems
• End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning
• CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researcher
• Learning Retrieval Augmentation for Personalized Dialogue Generation
• On Early Detection of Hallucinations in Factual Question Answering
• Language Models As or For Knowledge Bases
• Enhancing Tourism Recommender Systems for Sustainable City Trips Using Retrieval-Augmented Generation
• REX: Designing User-centered Repair and Explanations to Address Robot Failures
• Neural Machine Translation with Monolingual Translation Memory
• LLM4PM: A case study on using Large Language Models for Process Modeling in Enterprise Organizations
• Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL
• Pre-Trained Models: Past, Present and Future
• Ethel: A Virtual Teaching Assistant
• FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion
• Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models
• Search-Engine-augmented Dialogue Response Generation with Cheaply Supervised Query Production
• Large Language Models Are Unreliable for Cyber Threat Intelligence
• Automating Traffic Model Enhancement with AI Research Agent
• Retrieval-Augmented Audio Deepfake Detection
• The Life Cycle of Knowledge in Big Language Models: A Survey
• Knowing You Don't Know: Learning When to Continue Search in Multi-round RAG through Self-Practicing
• APOLLO: A GPT-based tool to detect phishing emails and generate explanations that warn users
• LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot
• Explainable Recommendation with Personalized Review Retrieval and Aspect Learning
• MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
• Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG
• QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation
• Walert: Putting Conversational Search Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot
• On Sarcasm Detection with OpenAI GPT-based Models
• Attention-guided Generative Models for Extractive Question Answering
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• Securing RAG: A Risk Assessment and Mitigation Framework
• Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
• Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets
• Agents for self-driving laboratories applied to quantum computing
• AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles
• Multi-view Intent Learning and Alignment with Large Language Models for Session-based Recommendation
• Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System
• Deanthropomorphising NLP: Can a Language Model Be Conscious?
• LAMBDA: A Large Model Based Data Agent
• iRAG: Advancing RAG for Videos with an Incremental Approach
• Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning
• Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
• A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options
• Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation
• Truthful AI: Developing and governing AI that does not lie
• Teola: Towards End-to-End Optimization of LLM-based Applications
• Logic Augmented Generation
• How Managers Perceive AI-Assisted Conversational Training for Workplace Communication
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Open-book Video Captioning with Retrieve-Copy-Generate Network
• DARTH: Declarative Recall Through Early Termination for Approximate Nearest Neighbor Search
• A Study of Situational Reasoning for Traffic Understanding
• Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
• LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes
• Development of Application-Specific Large Language Models to Facilitate Research Ethics Review
• SlideItRight: Using AI to Find Relevant Slides and Provide Feedback for Open-Ended Questions
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
• Fast Graph Vector Search via Hardware Acceleration and Delayed-Synchronization Traversal
• Autograding Mathematical Induction Proofs with Natural Language Processing
• DEEPAGÉ: Answering Questions in Portuguese about the Brazilian Environment
• ARCeR: an Agentic RAG for the Automated Definition of Cyber Ranges
• Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation
• Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
• Integrating question answering and text-to-SQL in Portuguese
• Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese
• Knowledge Graph-Driven Retrieval-Augmented Generation: Integrating Deepseek-R1 with Weaviate for Advanced Chatbot Applications
• FathomGPT: A Natural Language Interface for Interactively Exploring Ocean Science Data
• The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants
• Data Processing with FPGAs on Modern Architectures
• Cracking Vector Search Indexes
• Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes
• R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models
• Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguation
• Predictive Modeling: BIM Command Recommendation Based on Large-scale Usage Logs
• Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
• FDM-Bench: A Comprehensive Benchmark for Evaluating Large Language Models in Additive Manufacturing Tasks
• Learning Dense Representations of Phrases at Scale
• Multimedia Generative Script Learning for Task Planning
• Opportunities and Challenges in Harnessing Digital Technology for Effective Teaching and Learning
• Efficient Domain Adaptation of Sentence Embeddings Using Adapters
• Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
• Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
• ESG Accountability Made Easy: DocQA at Your Service
• Bias-Aware Agent: Enhancing Fairness in AI-Driven Knowledge Retrieval
• PEEK: A Large Dataset of Learner Engagement with Educational Videos
• A Self-supervised Method for Entity Alignment
• Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-Answering
• BibSonomy Meets ChatLLMs for Publication Management: From Chat to Publication Management: Organizing your related work using BibSonomy & LLMs
• Physio: An LLM-Based Physiotherapy Advisor
• ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering
• Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024
• CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
• GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
• GRASP: Municipal Budget AI Chatbots for Enhancing Civic Engagement
• EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering
• Optimizing the role of human evaluation in LLM-based spoken document summarization systems
• Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection
• K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• AutoMathKG: The automated mathematical knowledge graph based on LLM and vector database
• AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations
• Towards A Multi-agent System for Online Hate Speech Detection
• Opportunistically Parallel Lambda Calculus
• Bridging Text and Crystal Structures: Literature-driven Contrastive Learning for Materials Science
• Can open source large language models be used for tumor documentation in Germany? -- An evaluation on urological doctors' notes
• BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
• Bingo: Radix-based Bias Factorization for Random Walk on Dynamic Graphs
• Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
• Gen-AI for User Safety: A Survey
• LogBabylon: A Unified Framework for Cross-Log File Integration and Analysis
• Exploiting Pre-trained Models for Drug Target Affinity Prediction with Nearest Neighbors
• DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Exploration
• Current Limitations of Language Models: What You Need is Retrieval
• TempEL: Linking Dynamically Evolving and Newly Emerging Entities
• Thrill-K Architecture: Towards a Solution to the Problem of Knowledge Based Understanding
• Which Discriminator for Cooperative Text Generation?
• Should we tweet this? Generative response modeling for predicting reception of public health messaging on Twitter
• Conversational Answer Generation and Factuality for Reading Comprehension Question-Answering
• Towards Robust Neural Retrieval Models with Synthetic Pre-Training
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
• The benefits of query-based KGQA systems for complex and temporal questions in LLM era
• Latent Sensor Fusion: Multimedia Learning of Physiological Signals for Resource-Constrained Devices
• Variational Learning for Unsupervised Knowledge Grounded Dialogs
• Adopting Large Language Models to Automated System Integration
• Valid Text-to-SQL Generation with Unification-based DeepStochLog
• Simplifying Data Integration: SLM-Driven Systems for Unified Semantic Queries Across Heterogeneous Databases
• Dense Hierarchical Retrieval for Open-Domain Question Answering
• Network-informed Prompt Engineering against Organized Astroturf Campaigns under Extreme Class Imbalance
• Local Knowledge Powered Conversational Agents
• Towards Repository-Level Program Verification with Large Language Models
• Automatic Claim Review for Climate Science via Explanation Generation
• Zero-shot Slot Filling with DPR and RAG
• Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization
• Towards Standards-Compliant Assistive Technology Product Specifications via LLMs
• Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
• Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A
• CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation
• Encapsulating Textual Contents into a MOC data Structure for Advanced Applications
• Learning to Selectively Learn for Weakly-supervised Paraphrase Generation
• Recall and Learn: A Memory-augmented Solver for Math Word Problems
• End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated Approach
• Towards Faster k-Nearest-Neighbor Machine Translation
• Reliable Conversational Agents under ASP Control that Understand Natural Language
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
• Machine Reading Comprehension using Case-based Reasoning
• Instance-Based Neural Dependency Parsing
• Guided Decoding and Its Critical Role in Retrieval-Augmented Generation
• Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention
• Derivation Prompting: A Logic-Based Method for Improving Retrieval-Augmented Generation
• Real-Time Procedural Learning From Experience for AI Agents
• Efficient Retrieval Optimized Multi-task Learning
• TCSR-SQL: Towards Table Content-aware Text-to-SQL with Self-retrieval
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
• Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering
• Question Answering with Texts and Tables through Deep Reinforcement Learning
• Red Dragon AI at TextGraphs 2021 Shared Task: Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings
• Case-based Reasoning for Natural Language Queries over Knowledge Bases
• ClusterChat: Multi-Feature Search for Corpus Exploration
• Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
• Towards Universal Dense Retrieval for Open-domain Question Answering