Better Text Understanding Through Image-To-Text Transfer
arXiv:1705.08386
Abstract
Generic text embeddings are successfully used in a variety of tasks. However, they are often learnt by capturing the co-occurrence structure from pure text corpora, resulting in limitations of their ability to generalize. In this paper, we explore models that incorporate visual information into the text representation. Based on comprehensive ablation studies, we propose a conceptually simple, yet well performing architecture. It outperforms previous multimodal approaches on a set of well established benchmarks. We also improve the state-of-the-art results for image-related text datasets, using orders of magnitude less data.
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Towards Universal Paraphrastic Sentence Embeddings
- Order-Embeddings of Images and Language