Reciprocal Attention Fusion for Visual Question Answering
arXiv:1805.04247
Abstract
Existing attention mechanisms either attend to local image grid or object level features for Visual Question Answering (VQA). Motivated by the observation that questions can relate to both object instances and their parts, we propose a novel attention mechanism that jointly considers reciprocal relationships between the two levels of visual details. The bottom-up attention thus generated is further coalesced with the top-down information to only focus on the scene elements that are most relevant to a given question. Our design hierarchically fuses multi-modal information i.e., language, object- and gird-level features, through an efficient tensor decomposition scheme. The proposed model improves the state-of-the-art single model performances from 67.9% to 68.2% on VQAv1 and from 65.7% to 67.4% on VQAv2, demonstrating a significant boost.
To appear in the British Machine Vision Conference (BMVC), September 2018
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Aggregated Residual Transformations for Deep Neural Networks
- Dual Attention Networks for Multimodal Reasoning and Matching