Textually Enriched Neural Module Networks for Visual Question Answering
arXiv:1809.08697
Abstract
Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- ADADELTA: An Adaptive Learning Rate Method
- Caffe: Convolutional Architecture for Fast Feature Embedding
- NLTK: The Natural Language Toolkit
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Learning to Compose Neural Networks for Question Answering
- Image Labeling by Assignment
- Dual Attention Networks for Multimodal Reasoning and Matching
- Learning to count with deep object features