Multimodal Residual Learning for Visual QA

Abstract

Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

13 pages, 7 figures, accepted for NIPS 2016

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Dynamic Memory Networks for Visual and Textual Question Answering
• A Focused Dynamic Attention Model for Visual Question Answering

Cited by in corpus (46)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Residual Attention Network for Image Classification
• Modulating early visual processing by language
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Deep Multimodal Subspace Clustering Networks
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Multi-modal Deep Analysis for Multimedia
• GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
• Dual Attention Networks for Multimodal Reasoning and Matching
• Visual Question Answering: A Survey of Methods and Datasets
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Revisiting Visual Question Answering Baselines
• Text as Neural Operator: Image Manipulation by Text Instruction
• Structured Attentions for Visual Question Answering
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• Question-Guided Hybrid Convolution for Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• DualNet: Domain-Invariant Network for Visual Question Answering
• Visual Question Answering with Memory-Augmented Networks
• Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition
• Compact Trilinear Interaction for Visual Question Answering
• OmniNet: A unified architecture for multi-modal multi-task learning
• Dual Attention Networks for Visual Reference Resolution in Visual Dialog
• The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)
• I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion
• Facial age estimation by deep residual decision making
• Adaptive Offline Quintuplet Loss for Image-Text Matching
• Dynamic Filtering with Large Sampling Field for ConvNets
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• How to Become Instagram Famous: Post Popularity Prediction with Dual-Attention
• Visual Question Answering as Reading Comprehension
• Question Guided Modular Routing Networks for Visual Question Answering
• Recurrent Multimodal Interaction for Referring Image Segmentation
• Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction
• Conditional Transfer with Dense Residual Attention: Synthesizing traffic signs from street-view imagery