Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention

Abstract

The lifecycle of hallucination in LLMs is a concept that enables building solid frameworks on the control and reliability of LLMs in high-stakes environments, including health, legal, and scientific research. Although previous surveys have primarily focused on detection or mitigation, this survey provides a lifecycle-based overview of the hallucinations in the LLMs, their cause, detection, mitigation, and prevention.We propose a three-fold categorization of hallucinations across the LLM lifecycle: data-related, training-related, and inference-related, which is consistent with the lifecycle of the development of the LLM. Each of these stages is discussed regarding the cause of hallucinations, their detection, and the ways they can be addressed under specific mitigation or prevention interventions. In addition, we discuss the available benchmark data using a number of parameters so as to establish their suitability in identifying, restricting and managing hallucinations. The survey provides researchers and practitioners with a standardized framework to understand, diagnose, and cure hallucinations in a systematic system to present actionable data to build safer and more reliable LLMs.

References in corpus (35)

• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Towards Expert-Level Medical Question Answering with Large Language Models
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models
• A Survey of Hallucination in Large Foundation Models
• How Language Model Hallucinations Can Snowball
• Cognitive Mirage: A Review of Hallucinations in Large Language Models
• DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Improving Language Models via Plug-and-Play Retrieval Feedback
• Transformer-Patcher: One Mistake worth One Neuron
• Augmenting LLMs with Knowledge: A survey on hallucination prevention
• Zero-Resource Hallucination Prevention for Large Language Models
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models
• Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
• PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions
• Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability
• Improving Translation Faithfulness of Large Language Models via Augmenting Instructions
• BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer
• Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• Faithfulness-Aware Decoding Strategies for Abstractive Summarization
• Hallucination Improves the Performance of Unsupervised Visual Representation Learning
• Neural Knowledge Bank for Pretrained Transformers
• Mitigating Entity-Level Hallucination in Large Language Models
• mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations
• Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA
• Instruction Position Matters in Sequence Generation with Large Language Models
• "Sorry, Come Again?" Prompting -- Enhancing Comprehension and Diminishing Hallucination with [PAUSE]-injected Optimal Paraphrasing
• Entity Cloze By Date: What LMs Know About Unseen Entities
• Diving Deep into Modes of Fact Hallucinations in Dialogue Systems