Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
arXiv:1803.05268 · doi:10.1109/CVPR.2018.00519
Abstract
Visual question answering requires high-order reasoning about an image, which is a fundamental capability needed by machine systems to follow complex directives. Recently, modular networks have been shown to be an effective framework for performing visual reasoning tasks. While modular networks were initially designed with a degree of model transparency, their performance on complex visual reasoning benchmarks was lacking. Current state-of-the-art approaches do not provide an effective mechanism for understanding the reasoning process. In this paper, we close the performance gap between interpretable models and state-of-the-art visual reasoning methods. We propose a set of visual-reasoning primitives which, when composed, manifest as a model capable of performing complex reasoning tasks in an explicitly-interpretable manner. The fidelity and interpretability of the primitives' outputs enable an unparalleled ability to diagnose the strengths and weaknesses of the resulting model. Critically, we show that these primitives are highly performant, achieving state-of-the-art accuracy of 99.1% on the CLEVR dataset. We also show that our model is able to effectively learn generalized representations when provided a small amount of data containing novel object attributes. Using the CoGenT generalization task, we show more than a 20 percentage point improvement over the current state of the art.
arXiv commentCVPR 2018 pre-print
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- A simple neural network module for relational reasoning
- Residual Attention Network for Image Classification
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Learning to Count Objects in Natural Images for Visual Question Answering
- FiLM: Visual Reasoning with a General Conditioning Layer
- Stacked Attention Networks for Image Question Answering
- Compositional Attention Networks for Machine Reasoning
- Learning to Compose Neural Networks for Question Answering
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- Video Summarization with Attention-Based Encoder-Decoder Networks
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
- Learning Visual Reasoning Without Strong Priors
- Sensor Transformation Attention Networks
Cited by in corpus (50)
- From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
- Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
- Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
- A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
- Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning
- Multi-modal Deep Analysis for Multimedia
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- CLOSURE: Assessing Systematic Generalization of CLEVR Models
- A Comprehensive Survey of Machine Learning Applied to Radar Signal Processing
- RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
- Discrete and continuous representations and processing in deep learning: Looking forward
- MUREL: Multimodal Relational Reasoning for Visual Question Answering
- Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games
- Solving the Same-Different Task with Convolutional Neural Networks
- Learning by Abstraction: The Neural State Machine
- Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
- Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning
- Learning from Lexical Perturbations for Consistent Visual Question Answering
- Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
- Learning Visual Question Answering by Bootstrapping Hard Attention
- Interpretable Visual Question Answering by Reasoning on Dependency Trees
- On transfer learning using a MAC model variant
- Answer Them All! Toward Universal Visual Question Answering Models
- Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
- HySTER: A Hybrid Spatio-Temporal Event Reasoner
- How to Design Sample and Computationally Efficient VQA Models
- Graph Neural Networks for Image Understanding Based on Multiple Cues: Group Emotion Recognition and Event Recognition as Use Cases
- Generating Rationales in Visual Question Answering
- Visual Superordinate Abstraction for Robust Concept Learning
- Linguistically Driven Graph Capsule Network for Visual Question Reasoning
- Question Guided Modular Routing Networks for Visual Question Answering
- AiR: Attention with Reasoning Capability
- Cascaded Mutual Modulation for Visual Reasoning
- Transfer Learning in Visual and Relational Reasoning
- Explainable Neural Computation via Stack Neural Module Networks
- Hopper: Multi-hop Transformer for Spatiotemporal Reasoning
- What is needed for simple spatial language capabilities in VQA?
- Neuro-Symbolic VQA: A review from the perspective of AGI desiderata
- Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention
- Automatic Documentation of ICD Codes with Far-Field Speech Recognition
- ORGAN: Object-Centric Representation Learning using Cycle Consistent Generative Adversarial Networks
- Disentanglement-Based Equivariant Learning for Compositional VQA
- Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization
- Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
- Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering
- Recognizing Part Attributes with Insufficient Data
- Domain-robust VQA with diverse datasets and methods but no target labels
- Interaction Design for Explainable AI: Workshop Proceedings
- Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network