Question-Guided Hybrid Convolution for Visual Question Answering

Abstract

In this paper, we propose a novel Question-Guided Hybrid Convolution (QGHC) network for Visual Question Answering (VQA). Most state-of-the-art VQA methods fuse the high-level textual and visual features from the neural network and abandon the visual spatial information when learning multi-modal features.To address these problems, question-guided kernels generated from the input question are designed to convolute with visual features for capturing the textual and visual relationship in the early stage. The question-guided convolution can tightly couple the textual and visual information but also introduce more parameters when learning kernels. We apply the group convolution, which consists of question-independent kernels and question-dependent kernels, to reduce the parameter size and alleviate over-fitting. The hybrid convolution can generate discriminative multi-modal features with fewer parameters. The proposed approach is also complementary to existing bilinear pooling fusion and attention based VQA methods. By integrating with them, our method could further boost the performance. Extensive experiments on public VQA datasets validate the effectiveness of QGHC.

arXiv comment17 pages, 4 figures, accepted in ECCV 2018

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Aggregated Residual Transformations for Deep Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• Modulating early visual processing by language
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• Learning to Compose Neural Networks for Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Dual Attention Networks for Multimodal Reasoning and Matching

Cited by in corpus (12)

• FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction
• Dynamic Neural Networks: A Survey
• Multi-modality Latent Interaction Network for Visual Question Answering
• PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Character Matters: Video Story Understanding with Character-Aware Relations
• DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
• A^2-Net: Molecular Structure Estimation from Cryo-EM Density Volumes
• PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
• Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
• Question Guided Modular Routing Networks for Visual Question Answering