Multi-modality Latent Interaction Network for Visual Question Answering

Abstract

Exploiting relationships between visual regions and question words have achieved great success in learning multi-modality features for Visual Question Answering (VQA). However, we argue that existing methods mostly model relations between individual visual regions and words, which are not enough to correctly answer the question. From humans' perspective, answering a visual question requires understanding the summarizations of visual and language information. In this paper, we proposed the Multi-modality Latent Interaction module (MLI) to tackle this problem. The proposed module learns the cross-modality relationships between latent visual and language summarizations, which summarize visual regions and question into a small number of latent representations to avoid modeling uninformative individual region-word relations. The cross-modality information between the latent summarizations are propagated to fuse valuable information from both modalities and are used to update the visual and word features. Such MLI modules can be stacked for several stages to model complex and latent relations between the two modalities and achieves highly competitive performance on public VQA benchmarks, VQA v2.0 and TDIUC . In addition, we show that the performance of our methods could be significantly improved by combining with pre-trained language model BERT.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Skip-Thought Vectors
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Learning to Count Objects in Natural Images for Visual Question Answering
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• Weakly-supervised Compositional FeatureAggregation for Few-shot Recognition
• Question-Guided Hybrid Convolution for Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• 2nd Place Solution to the GQA Challenge 2019

Cited by in corpus (14)

• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• End-to-End Object Detection with Adaptive Clustering Transformer
• Normalized and Geometry-Aware Self-Attention Network for Image Captioning
• Contrastive Visual-Linguistic Pretraining
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• Fast Convergence of DETR with Spatially Modulated Co-Attention
• Character Matters: Video Story Understanding with Character-Aware Relations
• Learning to Respond with Your Favorite Stickers: A Framework of Unifying Multi-Modality and User Preference in Multi-Turn Dialog
• SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering
• Greedy Gradient Ensemble for Robust Visual Question Answering
• Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
• Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog
• Extreme Low-Light Imaging with Multi-granulation Cooperative Networks
• What is Multimodality?