Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
arXiv:1808.00265
Abstract
A key aspect of VQA models that are interpretable is their ability to ground their answers to relevant regions in the image. Current approaches with this capability rely on supervised learning and human annotated groundings to train attention mechanisms inside the VQA architecture. Unfortunately, obtaining human annotations specific for visual grounding is difficult and expensive. In this work, we demonstrate that we can effectively train a VQA architecture with grounding supervision that can be automatically obtained from available region descriptions and object annotations. We also show that our model trained with this mined supervision generates visual groundings that achieve a higher correlation with respect to manually-annotated groundings, meanwhile achieving state-of-the-art VQA accuracy.
arXiv comment8 pages, 4 figures
References in corpus (7)
- NLTK: The Natural Language Toolkit
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
- Exploring Human-like Attention Supervision in Visual Question Answering
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation