Structured Attentions for Visual Question Answering

Abstract

Visual attention, which assigns weights to image regions according to their relevance to a question, is considered as an indispensable part by most Visual Question Answering models. Although the questions may involve complex relations among multiple regions, few attention models can effectively encode such cross-region relations. In this paper, we demonstrate the importance of encoding such relations by showing the limited effective receptive field of ResNet on two datasets, and propose to model the visual attention as a multivariate distribution over a grid-structured Conditional Random Field on image regions. We demonstrate how to convert the iterative inference algorithms, Mean Field and Loopy Belief Propagation, as recurrent layers of an end-to-end neural network. We empirically evaluated our model on 3 datasets, in which it surpasses the best baseline model of the newly released CLEVR dataset by 9.5%, and the best published model on the VQA dataset by 1.25%. Source code is available at https: //github.com/zhuchen03/vqa-sva.

ICCV2017

References in corpus (12)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Multimodal Residual Learning for Visual QA
• Hadamard Product for Low-rank Bilinear Pooling
• A Focused Dynamic Attention Model for Visual Question Answering
• Structured Attention Networks
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Dual Attention Networks for Multimodal Reasoning and Matching

Cited by in corpus (6)

• Learning to Count Objects in Natural Images for Visual Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures