Evaluating the Reliability of Self-Explanations in Large Language Models

Abstract

This paper investigates the reliability of explanations generated by large language models (LLMs) when prompted to explain their previous output. We evaluate two kinds of such self-explanations - extractive and counterfactual - using three state-of-the-art LLMs (2B to 8B parameters) on two different classification tasks (objective and subjective). Our findings reveal, that, while these self-explanations can correlate with human judgement, they do not fully and accurately follow the model's decision process, indicating a gap between perceived and actual model reasoning. We show that this gap can be bridged because prompting LLMs for counterfactual explanations can produce faithful, informative, and easy-to-verify results. These counterfactuals offer a promising alternative to traditional explainability methods (e.g. SHAP, LIME), provided that prompts are tailored to specific tasks and checked for validity.

Non peer-reviewed preprint. Presented at Discovery Science 2024. Peer-reviewed version published in the Springer Lecture Notes in Computer Science (vol 15243)

References in corpus (12)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• The Curious Case of Neural Text Degeneration
• Understanding Neural Networks through Representation Erasure
• Mistral 7B
• Gemma: Open Models Based on Gemini Research and Technology
• The Falcon Series of Open Language Models
• Interpretation of Prediction Models Using the Input Gradient
• Rationalization for Explainable NLP: A Survey
• Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations