Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations

Abstract

With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal representations. Specifically, our model attends to different types of textual semantics in two languages and visual objects for fine-grained alignments between sentences and images. We introduce a new objective function which explicitly encourages attention diversity to learn an improved visual-semantic embedding space. We evaluate our model in the German-Image and English-Image matching tasks on the Multi30K dataset, and in the Semantic Textual Similarity task with the English descriptions of visual content. Results show that our model yields a significant performance gain over other methods in all of the three tasks.

arXiv commentAccepted at EMNLP 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VQA: Visual Question Answering
• Imagination improves Multimodal Translation
• Order-Embeddings of Images and Language
• Stacked Cross Attention for Image-Text Matching
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
• Multilingual Multi-modal Embeddings for Natural Language Processing