Modulating and attending the source image during encoding improves Multimodal Translation
arXiv:1712.03449
Abstract
We propose a new and fully end-to-end approach for multimodal translation where the source text encoder modulates the entire visual input processing using conditional batch normalization, in order to compute the most informative image features for our task. Additionally, we propose a new attention mechanism derived from this original idea, where the attention model for the visual input is conditioned on the source text encoder representations. In the paper, we detail our models as well as the image analysis pipeline. Finally, we report experimental results. They are, as far as we know, the new state of the art on three different test sets.
arXiv commentAccepted at NIPS Workshop
References in corpus (7)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Layer Normalization
- Modulating early visual processing by language
- Imagination improves Multimodal Translation
- An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
- Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation
Cited by in corpus (5)
- UMONS Submission for WMT18 Multimodal Translation Task
- A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
- IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task
- Bringing back simplicity and lightliness into neural image captioning
- Adversarial reconstruction for Multi-modal Machine Translation