Visual hallucination detection in large vision-language models via evidential conflict

Abstract

Despite the remarkable multimodal capabilities of Large Vision-Language Models (LVLMs), discrepancies often occur between visual inputs and textual outputs--a phenomenon we term visual hallucination. This critical reliability gap poses substantial risks in safety-critical Artificial Intelligence (AI) applications, necessitating a comprehensive evaluation benchmark and effective detection methods. Firstly, we observe that existing visual-centric hallucination benchmarks mainly assess LVLMs from a perception perspective, overlooking hallucinations arising from advanced reasoning capabilities. We develop the Perception-Reasoning Evaluation Hallucination (PRE-HAL) dataset, which enables the systematic evaluation of both perception and reasoning capabilities of LVLMs across multiple visual semantics, such as instances, scenes, and relations. Comprehensive evaluation with this new benchmark exposed more visual vulnerabilities, particularly in the more challenging task of relation reasoning. To address this issue, we propose, to the best of our knowledge, the first Dempster-Shafer theory (DST)-based visual hallucination detection method for LVLMs through uncertainty estimation. This method aims to efficiently capture the degree of conflict in high-level features at the model inference phase. Specifically, our approach employs simple mass functions to mitigate the computational complexity of evidence combination on power sets. We conduct an extensive evaluation of state-of-the-art LVLMs, LLaVA-v1.5, mPLUG-Owl2 and mPLUG-Owl3, with the new PRE-HAL benchmark. Experimental results indicate that our method outperforms five baseline uncertainty metrics, achieving average AUROC improvements of 4%, 10%, and 7% across three LVLMs. Our code is available at https://github.com/HT86159/Evidential-Conflict.

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Survey of Hallucination in Natural Language Generation
• Flamingo: a Visual Language Model for Few-Shot Learning
• DINOv2: Learning Robust Visual Features without Supervision
• Visual Instruction Tuning
• Evidential Deep Learning to Quantify Classification Uncertainty
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Decision-Making with Belief Functions: a Review
• Language Models (Mostly) Know What They Know
• Logistic Regression, Neural Networks and Dempster-Shafer Theory: a New Perspective
• Lymphoma segmentation from 3D PET-CT images using a deep evidential network
• Teaching Models to Express Their Uncertainty in Words
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models