Generating Visual Explanations

Abstract

Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself. Existing approaches for deep visual recognition are generally opaque and do not output any justification text; contemporary vision-language models can describe image content but fail to take into account class-discriminative image aspects which justify visual predictions. We propose a new model that focuses on the discriminating properties of the visible object, jointly predicts a class label, and explains why the predicted label is appropriate for the image. We propose a novel loss function based on sampling and reinforcement learning that learns to generate sentences that realize a global sentence property, such as class specificity. Our results on a fine-grained bird species classification dataset show that our model is able to generate explanations which are not only consistent with an image but also more discriminative than descriptions produced by existing captioning methods.

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Explain Images with Multimodal Recurrent Neural Networks
• Learning to Compose Neural Networks for Question Answering
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• Guiding Long-Short Term Memory for Image Caption Generation
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Learning Discriminative Features via Label Consistent Neural Network

Cited by in corpus (37)

• What do we need to build explainable AI systems for the medical domain?
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences
• An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
• Discriminability objective for training descriptive captions
• Image as Data: Automated Visual Content Analysis for Political Science
• Inverse Classification for Comparison-based Interpretability in Machine Learning
• Stakeholders in Explainable AI
• Counterfactual Visual Explanations
• It Takes Two to Tango: Towards Theory of AI's Mind
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
• AI Research Considerations for Human Existential Safety (ARCHES)
• Context-aware Captions from Context-agnostic Supervision
• Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention
• Teaching Categories to Human Learners with Visual Explanations
• Consensus-based Sequence Training for Video Captioning
• InterpNET: Neural Introspection for Interpretable Deep Learning
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Textual Explanations for Self-Driving Vehicles
• Saliency-based Sequential Image Attention with Multiset Prediction
• Teaching Meaningful Explanations
• Do Explanations make VQA Models more Predictable to a Human?
• Learning to Act Properly: Predicting and Explaining Affordances from Images
• Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Understanding Convolutional Networks with APPLE : Automatic Patch Pattern Labeling for Explanation
• Learning to Disambiguate by Asking Discriminative Questions
• Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
• Grounding Visual Explanations (Extended Abstract)
• DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
• Using Program Induction to Interpret Transition System Dynamics
• Explaining Transition Systems through Program Induction
• Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"
• Understanding Misclassifications by Attributes