Visual Question Answering with Memory-Augmented Networks
arXiv:1707.04968
Abstract
In this paper, we exploit a memory-augmented neural network to predict accurate answers to visual questions, even when those answers occur rarely in the training set. The memory network incorporates both internal and external memory blocks and selectively pays attention to each training exemplar. We show that memory-augmented neural networks are able to maintain a relatively long-term memory of scarce training exemplars, which is important for visual question answering due to the heavy-tailed distribution of answers in a general VQA setting. Experimental results on two large-scale benchmark datasets show the favorable performance of the proposed algorithm with a comparison to state of the art.
arXiv commentCVPR 2018
References in corpus (19)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Effective Approaches to Attention-based Neural Machine Translation
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Simple Baseline for Visual Question Answering
- Adding Gradient Noise Improves Learning for Very Deep Networks
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Learning to Remember Rare Events
- Multimodal Residual Learning for Visual QA
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Deep Networks with Internal Selective Attention through Feedback Connections
- Training Recurrent Answering Units with Joint Loss Minimization for VQA
- Pointing the Unknown Words
- Hierarchical Memory Networks
- Recurrent Neural Networks with External Memory for Language Understanding