Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention

Abstract

The lifecycle of hallucination in LLMs is a concept that enables building solid frameworks on the control and reliability of LLMs in high-stakes environments, including health, legal, and scientific research. Although previous surveys have primarily focused on detection or mitigation, this survey provides a lifecycle-based overview of the hallucinations in the LLMs, their cause, detection, mitigation, and prevention.We propose a three-fold categorization of hallucinations across the LLM lifecycle: data-related, training-related, and inference-related, which is consistent with the lifecycle of the development of the LLM. Each of these stages is discussed regarding the cause of hallucinations, their detection, and the ways they can be addressed under specific mitigation or prevention interventions. In addition, we discuss the available benchmark data using a number of parameters so as to establish their suitability in identifying, restricting and managing hallucinations. The survey provides researchers and practitioners with a standardized framework to understand, diagnose, and cure hallucinations in a systematic system to present actionable data to build safer and more reliable LLMs.

References in corpus (80)

• Survey of Hallucination in Natural Language Generation
• LLaMA: Open and Efficient Foundation Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• The Curious Case of Neural Text Degeneration
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
• Towards Expert-Level Medical Question Answering with Large Language Models
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• Towards Understanding Sycophancy in Language Models
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Textbooks Are All You Need
• A Survey of Hallucination in Large Foundation Models
• Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
• Woodpecker: Hallucination Correction for Multimodal Large Language Models
• Reducing hallucination in structured outputs via Retrieval-Augmented Generation
• How Language Model Hallucinations Can Snowball
• Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
• Cognitive Mirage: A Review of Hallucinations in Large Language Models
• The Perils & Promises of Fact-checking with Large Language Models
• Factuality Enhanced Language Models for Open-Ended Text Generation
• Discovering Language Model Behaviors with Model-Written Evaluations
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• Why Does ChatGPT Fall Short in Providing Truthful Answers?
• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
• Memory-Based Model Editing at Scale
• Hallucination of Multimodal Large Language Models: A Survey
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection
• Med-HALT: Medical Domain Hallucination Test for Large Language Models
• Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources
• The Internal State of an LLM Knows When It's Lying
• Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication
• Evaluating Object Hallucination in Large Vision-Language Models
• In-Context Retrieval-Augmented Language Models
• Citation: A Key to Building Responsible and Accountable Large Language Models
• Improving Language Models via Plug-and-Play Retrieval Feedback
• Deduplicating Training Data Makes Language Models Better
• Zero-Resource Hallucination Prevention for Large Language Models
• Augmenting LLMs with Knowledge: A survey on hallucination prevention
• Transformer-Patcher: One Mistake worth One Neuron
• Metacognitive Prompting Improves Understanding in Large Language Models
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
• KL-Divergence Guided Temperature Sampling
• Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
• Contrastive Learning Reduces Hallucination in Conversations
• Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models
• R-Tuning: Instructing Large Language Models to Say `I Don't Know'
• Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability
• PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions
• Improving Translation Faithfulness of Large Language Models via Augmenting Instructions
• Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
• BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer
• Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities
• SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals
• Detecting and Preventing Hallucinations in Large Vision Language Models
• MSSC-BiMamba: Multimodal Sleep Stage Classification and Early Diagnosis of Sleep Disorders with Bidirectional Mamba
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• Knowledge Verification to Nip Hallucination in the Bud
• Neural Knowledge Bank for Pretrained Transformers
• Faithfulness-Aware Decoding Strategies for Abstractive Summarization
• Hallucination Improves the Performance of Unsupervised Visual Representation Learning
• Mitigating Entity-Level Hallucination in Large Language Models
• Alignment for Honesty
• Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering
• mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations
• Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA
• Instruction Position Matters in Sequence Generation with Large Language Models
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate
• Entity Cloze By Date: What LMs Know About Unseen Entities
• "Sorry, Come Again?" Prompting -- Enhancing Comprehension and Diminishing Hallucination with [PAUSE]-injected Optimal Paraphrasing
• The Cultural Gene of Large Language Models: A Study on the Impact of Cross-Corpus Training on Model Values and Biases
• Diving Deep into Modes of Fact Hallucinations in Dialogue Systems