Multimodal Unified Attention Networks for Vision-and-Language Interactions

Abstract

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.

arXiv comment11 pages, 7 figures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• VQA: Visual Question Answering
• Recurrent Models of Visual Attention
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Learning to Count Objects in Natural Images for Visual Question Answering
• Hadamard Product for Low-rank Bilinear Pooling
• Compositional Attention Networks for Machine Reasoning
• A Focused Dynamic Attention Model for Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering

Cited by in corpus (8)

• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech
• Deep Multimodal Neural Architecture Search
• A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer
• Siamese Tracking with Lingual Object Constraints
• Latent Alignment of Procedural Concepts in Multimodal Recipes
• Answer-checking in Context: A Multi-modal FullyAttention Network for Visual Question Answering
• A Multimodal Target-Source Classifier with Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects