Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering

Abstract

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multi-modal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multi-modal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a co-attention mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-the-art performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb.

ICCV 2017

References in corpus (21)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Multimodal Residual Learning for Visual QA
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Dual Attention Networks for Multimodal Reasoning and Matching
• Uncovering Temporal Context for Video Question and Answering
• FVQA: Fact-based Visual Question Answering
• DualNet: Domain-Invariant Network for Visual Question Answering
• Factorized Bilinear Models for Image Recognition

Cited by in corpus (15)

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Self-Attentive Associative Memory
• DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Zero-Shot Sketch-Image Hashing
• Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding
• Local Temporal Bilinear Pooling for Fine-grained Action Parsing
• Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network
• Cross-Dataset Adaptation for Visual Question Answering
• MoNet: Moments Embedding Network
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures