Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
arXiv:1704.00260
Abstract
An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular, the task of visual recognition is aligned to the task of visual question answering by forcing each to use the same word-region embeddings. We show this leads to greater inductive transfer from recognition to VQA than standard multitask learning. Visual recognition also improves, especially for categories that have relatively few recognition training labels but appear often in the VQA setting. Thus, our paper takes a small step towards creating more general vision systems by showing the benefit of interpretable, flexible, and trainable core representations.
arXiv commentAccepted in ICCV 2017. The arxiv version has an extra analysis on correlation with human attention
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Efficient Estimation of Word Representations in Vector Space
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Teaching Machines to Read and Comprehend
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Recurrent Models of Visual Attention
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- A Focused Dynamic Attention Model for Visual Question Answering
- Attention Correctness in Neural Image Captioning