Learning Convolutional Text Representations for Visual Question Answering
arXiv:1705.06824 · doi:10.1137/1.9781611975321.67
Abstract
Visual question answering is a recently proposed artificial intelligence task that requires a deep understanding of both images and texts. In deep learning, images are typically modeled through convolutional neural networks, and texts are typically modeled through recurrent neural networks. While the requirement for modeling images is similar to traditional computer vision tasks, such as object recognition and image classification, visual question answering raises a different need for textual representation as compared to other natural language processing tasks. In this work, we perform a detailed analysis on natural language questions in visual question answering. Based on the analysis, we propose to rely on convolutional neural networks for learning textual representations. By exploring the various properties of convolutional neural networks specialized for text data, such as width and depth, we present our "CNN Inception + Gate" model. We show that our model improves question representations and thus the overall accuracy of visual question answering models. We also show that the text representation requirement in visual question answering is more complicated and comprehensive than that in conventional natural language processing tasks, making it a better task to evaluate textual representation methods. Shallow models like fastText, which can obtain comparable results with deep learning models in tasks like text classification, are not suitable in visual question answering.
arXiv commentConference paper at SDM 2018. https://github.com/divelab/svae
References in corpus (12)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Character-level Convolutional Networks for Text Classification
- Conditional Image Generation with PixelCNN Decoders
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Language Modeling with Gated Convolutional Networks
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Effective Use of Word Order for Text Categorization with Convolutional Neural Networks