Multilingual Image Description with Neural Sequence Models

Abstract

In this paper we present an approach to multi-language image description bringing together insights from neural machine translation and neural image description. To create a description of an image for a given target language, our sequence generation models condition on feature vectors from the image, the description from the source language, and/or a multimodal vector computed over the image and a description in the source language. In image description experiments on the IAPR-TC12 dataset of images aligned with English and German sentences, we find significant and substantial improvements in BLEU4 and Meteor scores for models trained over multiple languages, compared to a monolingual baseline.

Under review as a conference paper at ICLR 2016

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server

Cited by in corpus (17)

• Imagination improves Multimodal Translation
• Multimodal Pivots for Image Caption Translation
• Fluency-Guided Cross-Lingual Image Captioning
• Does Multimodality Help Human and Machine for Translation and Image Captioning?
• Multimodal Attention for Neural Machine Translation
• Incorporating Global Visual Features into Attention-Based Neural Machine Translation
• Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
• Doubly Attentive Transformer Machine Translation
• Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation
• Doubly-Attentive Decoder for Multi-modal Neural Machine Translation
• Image Captioning and Classification of Dangerous Situations
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• A Correlational Encoder Decoder Architecture for Pivot Based Sequence Generation
• Zero-resource Machine Translation by Multimodal Encoder-decoder Network with Multimedia Pivot
• Neural Machine Translation: A Review and Survey
• Improving Captioning for Low-Resource Languages by Cycle Consistency
• Distilling Translations with Visual Awareness