An empirical study on the effectiveness of images in Multimodal Neural Machine Translation

Abstract

In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multimodal translation task (English, image to German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.

arXiv commentAccepted to EMNLP 2017

References in corpus (12)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Recurrent Models of Visual Attention
• DRAW: A Recurrent Neural Network For Image Generation
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Multiple Object Recognition with Visual Attention
• Grammar as a Foreign Language
• Does Multimodality Help Human and Machine for Translation and Image Captioning?
• Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation

Cited by in corpus (11)

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Dynamic Context-guided Capsule Network for Multimodal Machine Translation
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Modulating and attending the source image during encoding improves Multimodal Translation
• UMONS Submission for WMT18 Multimodal Translation Task
• Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
• A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
• Bringing back simplicity and lightliness into neural image captioning
• Visual Agreement Regularized Training for Multi-Modal Machine Translation
• Distilling Translations with Visual Awareness
• Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training