Deep Modular Co-Attention Networks for Visual Question Answering

Abstract

Visual Question Answering (VQA) requires a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective `co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-of-the-art. Our best single model delivers 70.63$\%$ overall accuracy on the test-dev set. Code is available at https://github.com/MILVLG/mcan-vqa.

Accepted at CVPR 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Hadamard Product for Low-rank Bilinear Pooling
• Dual Attention Networks for Multimodal Reasoning and Matching
• Training Deeper Neural Machine Translation Models with Transparent Attention

Cited by in corpus (10)

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Discrete and continuous representations and processing in deep learning: Looking forward
• Meta Module Network for Compositional Visual Reasoning
• Sub-Instruction Aware Vision-and-Language Navigation
• TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
• X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
• Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks
• On the Significance of Question Encoder Sequence Model in the Out-of-Distribution Performance in Visual Question Answering
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering
• Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition