Dual Recurrent Attention Units for Visual Question Answering
arXiv:1802.00209
Abstract
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA models. In this paper, we propose a recurrent attention mechanism and show its benefits compared to the traditional convolutional approach. We perform two ablation studies to evaluate recurrent attention. First, we introduce a baseline VQA model with visual attention and test the performance difference between convolutional and recurrent attention on the VQA 2.0 dataset. Secondly, we design an architecture for VQA which utilizes dual (textual and visual) Recurrent Attention Units (RAUs). Using this model, we show the effect of all possible combinations of recurrent and convolutional dual attention. Our single model outperforms the first place winner on the VQA 2016 challenge and to the best of our knowledge, it is the second best performing single model on the VQA 1.0 dataset. Furthermore, our model noticeably improves upon the winner of the VQA 2017 challenge. Moreover, we experiment replacing attention mechanisms in state-of-the-art models with our RAUs and show increased performance.
arXiv comment8 pages, 5 figures
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Methods for Interpreting and Understanding Deep Neural Networks
- A Deep Reinforced Model for Abstractive Summarization
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Recurrent Models of Visual Attention
- Dynamic Memory Networks for Visual and Textual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Training Recurrent Answering Units with Joint Loss Minimization for VQA