Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
arXiv:1606.01847
Abstract
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
arXiv commentAccepted to EMNLP 2016
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Sequence to Sequence Learning with Neural Networks
- Deep Residual Learning for Image Recognition
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Dynamic Memory Networks for Visual and Textual Question Answering
- Simple Baseline for Visual Question Answering
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Stacked Attention Networks for Image Question Answering
- A Focused Dynamic Attention Model for Visual Question Answering
- Learning to Compose Neural Networks for Question Answering
- Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
- Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
- Segmentation from Natural Language Expressions
Cited by in corpus (183)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- A simple neural network module for relational reasoning
- Modulating early visual processing by language
- RSVQA: Visual Question Answering for Remote Sensing Data
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Audiovisual SlowFast Networks for Video Recognition
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- Biomedical Question Answering: A Survey of Approaches and Challenges
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
- Deep Modular Co-Attention Networks for Visual Question Answering
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Visual Reference Resolution using Attention Memory for Visual Dialog
- Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
- Bilinear Attention Networks
- Training Recurrent Answering Units with Joint Loss Minimization for VQA
- Bilinear CNNs for Fine-grained Visual Recognition
- Person Search with Natural Language Description
- Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
- Zero-Shot Visual Question Answering
- Multi-modal Deep Analysis for Multimedia
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- Attention on Attention for Image Captioning
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- Modality-Agnostic Attention Fusion for visual search with text feedback
- C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- ShapeWorld - A new test methodology for multimodal language understanding
- Dual Attention Networks for Multimodal Reasoning and Matching
- High-Order Attention Models for Visual Question Answering
- Visual Question Answering: A Survey of Methods and Datasets
- Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
- Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis
- Factorized Multimodal Transformer for Multimodal Sequential Learning
- TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
- Identity-Aware Textual-Visual Matching with Latent Co-attention
- A Multimodal Memes Classification: A Survey and Open Research Issues
- DeepStory: Video Story QA by Deep Embedded Memory Networks
- An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
- Multi-modality Latent Interaction Network for Visual Question Answering
- Creativity: Generating Diverse Questions using Variational Autoencoders
- CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
- Revisiting Visual Question Answering Baselines
- Multimodal Transformer with Multi-View Visual Representation for Image Captioning
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- DVQA: Understanding Data Visualizations via Question Answering
- Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation
- Spatially Adaptive Computation Time for Residual Networks
- Motion-Appearance Co-Memory Networks for Video Question Answering
- Counterfactual VQA: A Cause-Effect Look at Language Bias
- MemexQA: Visual Memex Question Answering
- RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
- CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
- Question-Guided Hybrid Convolution for Visual Question Answering
- An Analysis of Visual Question Answering Algorithms
- Learning Two-Branch Neural Networks for Image-Text Matching Tasks
- Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
- Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
- End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
- Convolutional Self-Attention Networks
- In Defense of Grid Features for Visual Question Answering
- Contrastive Visual-Linguistic Pretraining
- Dynamic Computational Time for Visual Attention
- Learning Convolutional Text Representations for Visual Question Answering
- Comprehension-guided referring expressions
- Intermediate Deep Feature Compression: the Next Battlefield of Intelligent Sensing
- Learning to Evaluate Image Captioning
- DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning
- A Read-Write Memory Network for Movie Story Understanding
- AMC: Attention guided Multi-modal Correlation Learning for Image Search
- Visual Question Answering with Memory-Augmented Networks
- Robustness Analysis of Visual QA Models by Basic Questions
- Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
- MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
- Speech-Based Visual Question Answering
- Learning Visual Knowledge Memory Networks for Visual Question Answering
- Compact Trilinear Interaction for Visual Question Answering
- Visual Entailment Task for Visually-Grounded Language Learning
- Information Aggregation for Multi-Head Attention with Routing-by-Agreement
- LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
- Joint Image Captioning and Question Answering
- Ways of Conditioning Generative Adversarial Networks
- Reciprocal Attention Fusion for Visual Question Answering
- TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
- ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
- Image-Question-Answer Synergistic Network for Visual Dialog
- Supervised and Unsupervised Transfer Learning for Question Answering
- X-Linear Attention Networks for Image Captioning
- Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
- Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
- The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)
- VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
- An Empirical Evaluation of Visual Question Answering for Novel Objects
- Fooling Vision and Language Models Despite Localization and Attention Mechanism
- Survey of Recent Advances in Visual Question Answering
- Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
- Information Maximizing Visual Question Generation
- VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
- Learning Visual Question Answering by Bootstrapping Hard Attention
- A Real-time Global Inference Network for One-stage Referring Expression Comprehension
- Learning Cross-modal Context Graph for Visual Grounding
- Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
- A Study on Multimodal and Interactive Explanations for Visual Question Answering
- A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
- Video Highlight Prediction Using Audience Chat Reactions
- MarioQA: Answering Questions by Watching Gameplay Videos
- Figure Captioning with Reasoning and Sequence-Level Training
- Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
- A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
- Textually Enriched Neural Module Networks for Visual Question Answering
- Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
- Tensor Product Generation Networks for Deep NLP Modeling
- Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
- Improving Visual Question Answering by Referring to Generated Paragraph Captions
- Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
- Visual Question Answering on 360° Images
- Zero-Shot Transfer VQA Dataset
- Modality-Balanced Models for Visual Dialogue
- Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
- TallyQA: Answering Complex Counting Questions
- Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
- Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
- Multimodal Differential Network for Visual Question Generation
- Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
- Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
- Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning
- Object-Level Context Modeling For Scene Classification with Context-CNN
- Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
- Tutorial on Answering Questions about Images with Deep Learning
- Fine-grained Video Categorization with Redundancy Reduction Attention
- Transformer for Emotion Recognition
- Modularized Textual Grounding for Counterfactual Resilience
- What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective
- MoNet: Moments Embedding Network
- Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
- Lessons learned in multilingual grounded language learning
- Textbook Question Answering with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension
- Cross-Dataset Adaptation for Visual Question Answering
- Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network
- Which Ads to Show? Advertisement Image Assessment with Auxiliary Information via Multi-step Modality Fusion
- Disentangled Motif-aware Graph Learning for Phrase Grounding
- Query-controllable Video Summarization
- Visual Question Answering as Reading Comprehension
- Local Temporal Bilinear Pooling for Fine-grained Action Parsing
- The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
- Geometry-constrained Car Recognition Using a 3D Perspective Network
- Question Guided Modular Routing Networks for Visual Question Answering
- Improved RAMEN: Towards Domain Generalization for Visual Question Answering
- The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
- Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
- A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
- Achieving Human Parity on Visual Question Answering
- Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
- An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis
- Enforcing Reasoning in Visual Commonsense Reasoning
- Granular Multimodal Attention Networks for Visual Dialog
- PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
- Transfer Learning in Visual and Relational Reasoning
- Deep Exemplar Networks for VQA and VQG
- Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
- Deep Multi-Modal Sets
- Recurrent Multimodal Interaction for Referring Image Segmentation
- Visual Explanations from Hadamard Product in Multimodal Deep Networks
- Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
- Guessing State Tracking for Visual Dialogue
- Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
- Uncertainty based Class Activation Maps for Visual Question Answering
- Adversarial Multimodal Network for Movie Question Answering
- Proposing Plausible Answers for Open-ended Visual Question Answering
- Approximated Bilinear Modules for Temporal Modeling
- Adventurer's Treasure Hunt: A Transparent System for Visually Grounded Compositional Visual Question Answering based on Scene Graphs
- Video SemNet: Memory-Augmented Video Semantic Network
- Pay Attention to Those Sets! Learning Quantification from Images
- Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture
- Dynamic Attention Networks for Task Oriented Grounding
- Parallel Intent and Slot Prediction using MLB Fusion
- CS-VQA: Visual Question Answering with Compressively Sensed Images
- Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images