Dual Attention Networks for Multimodal Reasoning and Matching

Abstract

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Dynamic Memory Networks for Visual and Textual Question Answering
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Stacked Attention Networks for Image Question Answering
• Show and Tell: A Neural Image Caption Generator
• Deep Networks with Internal Selective Attention through Feedback Connections
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Cited by in corpus (29)

• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Deep Modular Co-Attention Networks for Visual Question Answering
• Bilinear Attention Networks
• Discriminability objective for training descriptive captions
• High-Order Attention Models for Visual Question Answering
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Multimodal Transformer with Multi-View Visual Representation for Image Captioning
• Structured Attentions for Visual Question Answering
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• Question-Guided Hybrid Convolution for Visual Question Answering
• Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition
• Speech-Based Visual Question Answering
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images
• Reciprocal Attention Fusion for Visual Question Answering
• Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• Image-Question-Answer Synergistic Network for Visual Dialog
• Survey of Recent Advances in Visual Question Answering
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Textually Enriched Neural Module Networks for Visual Question Answering
• Zero-Shot Transfer VQA Dataset
• Automatic Music Highlight Extraction using Convolutional Recurrent Attention Networks
• Visual Explanations from Hadamard Product in Multimodal Deep Networks
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• Applying recent advances in Visual Question Answering to Record Linkage