Supervised Visual Attention for Simultaneous Multimodal Machine Translation
arXiv:2201.09324 · doi:10.1613/jair.1.13546
Abstract
Recently, there has been a surge in research in multimodal machine translation (MMT), where additional modalities such as images are used to improve translation quality of textual systems. A particular use for such multimodal systems is the task of simultaneous machine translation, where visual context has been shown to complement the partial information provided by the source sentence, especially in the early phases of translation. In this paper, we propose the first Transformer-based simultaneous MMT architecture, which has not been previously explored in the field. Additionally, we extend this model with an auxiliary supervision signal that guides its visual attention mechanism using labelled phrase-region alignments. We perform comprehensive experiments on three language directions and conduct thorough quantitative and qualitative analyses using both automatic metrics and manual inspection. Our results show that (i) supervised visual attention consistently improves the translation quality of the MMT models, and (ii) fine-tuning the MMT with supervision loss enabled leads to better performance than training the MMT from scratch. Compared to the state-of-the-art, our proposed model achieves improvements of up to 2.3 BLEU and 3.5 METEOR points.
arXiv commentAccepted to Journal of Artificial Intelligence Research (JAIR)
References in corpus (9)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Neural Machine Translation by Jointly Learning to Align and Translate
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Learning Deep Transformer Models for Machine Translation
- Does Multimodality Help Human and Machine for Translation and Image Captioning?
- Towards Multimodal Simultaneous Neural Machine Translation
- Phrase Localization Without Paired Training Examples