Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
arXiv:2001.01037
Abstract
This paper analyzes the predictions of image captioning models with attention mechanisms beyond visualizing the attention itself. We develop variants of layer-wise relevance propagation (LRP) and gradient-based explanation methods, tailored to image captioning models with attention mechanisms. We compare the interpretability of attention heatmaps systematically against the explanations provided by explanation methods such as LRP, Grad-CAM, and Guided Grad-CAM. We show that explanation methods provide simultaneously pixel-wise image explanations (supporting and opposing pixels of the input image) and linguistic explanations (supporting and opposing words of the preceding sequence) for each word in the predicted captions. We demonstrate with extensive experiments that explanation methods 1) can reveal additional evidence used by the model to make decisions compared to attention; 2) correlate to object locations with high precision; 3) are helpful to "debug" the model, e.g. by analyzing the reasons for hallucinated object words. With the observed properties of explanations, we further design an LRP-inference fine-tuning strategy that reduces the issue of object hallucination in image captioning models, and meanwhile, maintains the sentence fluency. We conduct experiments with two widely used attention mechanisms: the adaptive attention mechanism calculated with the additive attention and the multi-head attention mechanism calculated with the scaled dot product.
References in corpus (17)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Axiomatic Attribution for Deep Networks
- BERTScore: Evaluating Text Generation with BERT
- Learning Important Features Through Propagating Activation Differences
- Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
- Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
- Sanity Checks for Saliency Maps
- Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
- Learning how to explain neural networks: PatternNet and PatternAttribution
- RISE: Randomized Input Sampling for Explanation of Black-box Models
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Context-Aware Visual Policy Network for Fine-Grained Image Captioning
- From Clustering to Cluster Explanations via Neural Networks
- Cascaded Revision Network for Novel Object Captioning
- Adaptively Aligned Image Captioning via Adaptive Attention Time
- Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
- RelEx: A Model-Agnostic Relational Model Explainer