Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
arXiv:1511.05756
Abstract
We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions. For the adaptive parameter prediction, we employ a separate parameter prediction network, which consists of gated recurrent unit (GRU) taking a question as its input and a fully-connected layer generating a set of candidate weights as its output. However, it is challenging to construct a parameter prediction network for a large number of parameters in the fully-connected dynamic parameter layer of the CNN. We reduce the complexity of this problem by incorporating a hashing technique, where the candidate weights given by the parameter prediction network are selected using a predefined hash function to determine individual weights in the dynamic parameter layer. The proposed network---joint network with the CNN for ImageQA and the parameter prediction network---is trained end-to-end through back-propagation, where its weights are initialized using a pre-trained CNN and GRU. The proposed algorithm illustrates the state-of-the-art performance on all available public ImageQA benchmarks.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Going Deeper with Convolutions
- VQA: Visual Question Answering
- Predicting Parameters in Deep Learning
- Compressing Neural Networks with the Hashing Trick
- Verb Semantics and Lexical Selection
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions
Cited by in corpus (17)
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Multimodal Residual Learning for Visual QA
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Dual Attention Networks for Multimodal Reasoning and Matching
- Revisiting Video Saliency: A Large-scale Benchmark and a New Model
- Text as Neural Operator: Image Manipulation by Text Instruction
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
- Task-driven Visual Saliency and Attention-based Visual Question Answering
- An Analysis of Visual Question Answering Algorithms
- Learning Visual Knowledge Memory Networks for Visual Question Answering
- Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
- An Empirical Evaluation of Visual Question Answering for Novel Objects
- MarioQA: Answering Questions by Watching Gameplay Videos
- Learning to Disambiguate by Asking Discriminative Questions
- Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization
- The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
- Composing Text and Image for Image Retrieval - An Empirical Odyssey