Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation

Abstract

In state-of-the-art Neural Machine Translation, an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions. Approaches to pool two modalities usually include element-wise product, sum or concatenation. In this paper, we evaluate the more advanced Multimodal Compact Bilinear pooling method, which takes the outer product of two vectors to combine the attention features for the two modalities. This has been previously investigated for visual question answering. We try out this approach for multimodal image caption translation and show improvements compared to basic combination methods.

Submitted to ICLR Workshop 2017

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Does Multimodality Help Human and Machine for Translation and Image Captioning?
• Doubly-Attentive Decoder for Multi-modal Neural Machine Translation

Cited by in corpus (6)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
• Modulating and attending the source image during encoding improves Multimodal Translation
• Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
• Adversarial reconstruction for Multi-modal Machine Translation
• Neuron Interaction Based Representation Composition for Neural Machine Translation