Multimodal Attention for Neural Machine Translation
arXiv:1609.03976
Abstract
The attention mechanism is an important part of the neural machine translation (NMT) where it was reported to produce richer source representation compared to fixed-length encoding sequence-to-sequence models. Recently, the effectiveness of attention has also been explored in the context of image captioning. In this work, we assess the feasibility of a multimodal attention mechanism that simultaneously focus over an image and its natural language description for generating a description in another language. We train several variants of our proposed attention mechanism on the Multi30k multilingual image captioning dataset. We show that a dedicated attention for each modality achieves up to 1.6 points in BLEU and METEOR compared to a textual NMT baseline.
10 pages, under review COLING 2016
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Machine Translation by Jointly Learning to Align and Translate
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Image Captioning with Semantic Attention
- Multilingual Image Description with Neural Sequence Models
Cited by in corpus (9)
- Does Neural Machine Translation Benefit from Larger Context?
- Emergent Translation in Multi-Agent Communication
- VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
- Visual Cues and Error Correction for Translation Robustness
- MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
- Generative Imagination Elevates Machine Translation
- Multimodal Data Fusion based on the Global Workspace Theory
- Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than Character Level
- Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training