MUREL: Multimodal Relational Reasoning for Visual Question Answering

Abstract

Multimodal attentional networks are currently state-of-the-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks. In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question and image regions by a rich vectorial representation, and modeling region relations with pairwise combinations. Secondly, we incorporate the cell into a full MuRel network, which progressively refines visual and question interactions, and can be leveraged to define visualization schemes finer than mere attention maps. We validate the relevance of our approach with various ablation studies, and show its superiority to attention-based methods on three datasets: VQA 2.0, VQA-CP v2 and TDIUC. Our final MuRel network is competitive to or outperforms state-of-the-art results in this challenging context. Our code is available: https://github.com/Cadene/murel.bootstrap.pytorch

CVPR2019 accepted paper

References in corpus (10)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Skip-Thought Vectors
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Hadamard Product for Low-rank Bilinear Pooling
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Measuring abstract reasoning in neural networks
• Training Recurrent Answering Units with Joint Loss Minimization for VQA

Cited by in corpus (11)

• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Meta Module Network for Compositional Visual Reasoning
• Learning Cross-modal Context Graph for Visual Grounding
• 3DRM:Pair-wise relation module for 3D object detection
• Location-aware Graph Convolutional Networks for Video Question Answering
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• 3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video