Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering

Abstract

Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fusing multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering. We also show that the proposed dynamic intra-modality attention flow conditioned on the other modality can dynamically modulate the intra-modality attention of the target modality, which is vital for multimodality feature fusion. Experimental evaluations on the VQA 2.0 dataset show that the proposed method achieves state-of-the-art VQA performance. Extensive ablation studies are carried out for the comprehensive analysis of the proposed method.

CVPR 2019 ORAL

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional Sequence to Sequence Learning
• VQA: Visual Question Answering
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Dynamic Coattention Networks For Question Answering
• Simple Baseline for Visual Question Answering
• Learning to Count Objects in Natural Images for Visual Question Answering
• Hadamard Product for Low-rank Bilinear Pooling
• Structured Attention Networks
• Bilinear Attention Networks
• Question-Guided Hybrid Convolution for Visual Question Answering
• A^2-Net: Molecular Structure Estimation from Cryo-EM Density Volumes

Cited by in corpus (6)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Multi-modality Latent Interaction Network for Visual Question Answering
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review