Lessons learned in multilingual grounded language learning
arXiv:1809.07615
Abstract
Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language. Here, we investigate in detail which conditions affect the performance of this type of grounded language learning model. We show that multilingual training improves over bilingual training, and that low-resource languages benefit from training with higher-resource languages. We demonstrate that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language enables further improvements via an additional caption-caption ranking objective.
arXiv commentCoNLL 2018
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Generative Adversarial Text to Image Synthesis
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VQA: Visual Question Answering
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Imagination improves Multimodal Translation
- Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
- Order-Embeddings of Images and Language
- Revisiting Visual Question Answering Baselines
- Multilingual Multi-modal Embeddings for Natural Language Processing
- Improving Visually Grounded Sentence Representations with Self-Attention
- Continuous multilinguality with language vectors