Incorporating Global Visual Features into Attention-Based Neural Machine Translation

Abstract

We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. We utilise global image features extracted using a pre-trained convolutional neural network and incorporate them (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate how these different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal models. We report new state-of-the-art results and our best models also significantly improve on a comparable phrase-based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.

8 pages (11 including references), 5 figures

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Multilingual Image Description with Neural Sequence Models

Cited by in corpus (13)

• Imagination improves Multimodal Translation
• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
• Exploiting Cross-Sentence Context for Neural Machine Translation
• Gumbel-Attention for Multi-modal Machine Translation
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
• A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
• MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
• A Corpus for English-Japanese Multimodal Neural Machine Translation with Comparable Sentences
• Generative Imagination Elevates Machine Translation
• Video Question Generation via Cross-Modal Self-Attention Networks Learning
• Visual Agreement Regularized Training for Multi-Modal Machine Translation
• Probing Representations Learned by Multimodal Recurrent and Transformer Models