Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Abstract

Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.

Accepted to EMNLP 2016

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• A Focused Dynamic Attention Model for Visual Question Answering
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
• Segmentation from Natural Language Expressions

Cited by in corpus (181)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• A simple neural network module for relational reasoning
• Modulating early visual processing by language
• RSVQA: Visual Question Answering for Remote Sensing Data
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Audiovisual SlowFast Networks for Video Recognition
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Biomedical Question Answering: A Survey of Approaches and Challenges
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Deep Modular Co-Attention Networks for Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Bilinear Attention Networks
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Person Search with Natural Language Description
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Zero-Shot Visual Question Answering
• Multi-modal Deep Analysis for Multimedia
• Attention on Attention for Image Captioning
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Modality-Agnostic Attention Fusion for visual search with text feedback
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Dual Attention Networks for Multimodal Reasoning and Matching
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• ShapeWorld - A new test methodology for multimodal language understanding
• High-Order Attention Models for Visual Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Polarimetric Thermal to Visible Face Verification via Attribute Preserved Synthesis
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• A Multimodal Memes Classification: A Survey and Open Research Issues
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• DeepStory: Video Story QA by Deep Embedded Memory Networks
• Multi-modality Latent Interaction Network for Visual Question Answering
• Creativity: Generating Diverse Questions using Variational Autoencoders
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Multimodal Transformer with Multi-View Visual Representation for Image Captioning
• Revisiting Visual Question Answering Baselines
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• DVQA: Understanding Data Visualizations via Question Answering
• Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation
• Spatially Adaptive Computation Time for Residual Networks
• CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
• MemexQA: Visual Memex Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Counterfactual VQA: A Cause-Effect Look at Language Bias
• RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
• Question-Guided Hybrid Convolution for Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Convolutional Self-Attention Networks
• End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Contrastive Visual-Linguistic Pretraining
• In Defense of Grid Features for Visual Question Answering
• Dynamic Computational Time for Visual Attention
• Learning Convolutional Text Representations for Visual Question Answering
• Comprehension-guided referring expressions
• Intermediate Deep Feature Compression: the Next Battlefield of Intelligent Sensing
• Learning to Evaluate Image Captioning
• DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning
• Visual Question Answering with Memory-Augmented Networks
• Robustness Analysis of Visual QA Models by Basic Questions
• A Read-Write Memory Network for Movie Story Understanding
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• AMC: Attention guided Multi-modal Correlation Learning for Image Search
• Compact Trilinear Interaction for Visual Question Answering
• Speech-Based Visual Question Answering
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Visual Entailment Task for Visually-Grounded Language Learning
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
• TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
• Ways of Conditioning Generative Adversarial Networks
• Joint Image Captioning and Question Answering
• Reciprocal Attention Fusion for Visual Question Answering
• Supervised and Unsupervised Transfer Learning for Question Answering
• Image-Question-Answer Synergistic Network for Visual Dialog
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
• X-Linear Attention Networks for Image Captioning
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
• The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Information Maximizing Visual Question Generation
• Survey of Recent Advances in Visual Question Answering
• Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Learning Cross-modal Context Graph for Visual Grounding
• A Study on Multimodal and Interactive Explanations for Visual Question Answering
• A Real-time Global Inference Network for One-stage Referring Expression Comprehension
• Video Highlight Prediction Using Audience Chat Reactions
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Figure Captioning with Reasoning and Sequence-Level Training
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• Textually Enriched Neural Module Networks for Visual Question Answering
• MarioQA: Answering Questions by Watching Gameplay Videos
• Visual Question Answering on 360° Images
• Zero-Shot Transfer VQA Dataset
• Tensor Product Generation Networks for Deep NLP Modeling
• Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
• Improving Visual Question Answering by Referring to Generated Paragraph Captions
• Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
• Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• Modality-Balanced Models for Visual Dialogue
• TallyQA: Answering Complex Counting Questions
• Multimodal Differential Network for Visual Question Generation
• Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• Object-Level Context Modeling For Scene Classification with Context-CNN
• Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning
• Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
• Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network
• Disentangled Motif-aware Graph Learning for Phrase Grounding
• Query-controllable Video Summarization
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Modularized Textual Grounding for Counterfactual Resilience
• Question Guided Modular Routing Networks for Visual Question Answering
• Geometry-constrained Car Recognition Using a 3D Perspective Network
• Local Temporal Bilinear Pooling for Fine-grained Action Parsing
• Visual Question Answering as Reading Comprehension
• Textbook Question Answering with Multi-modal Context Graph Understanding and Self-supervised Open-set Comprehension
• Fine-grained Video Categorization with Redundancy Reduction Attention
• Lessons learned in multilingual grounded language learning
• Which Ads to Show? Advertisement Image Assessment with Auxiliary Information via Multi-step Modality Fusion
• Tutorial on Answering Questions about Images with Deep Learning
• What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective
• Cross-Dataset Adaptation for Visual Question Answering
• Transformer for Emotion Recognition
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
• MoNet: Moments Embedding Network
• Enforcing Reasoning in Visual Commonsense Reasoning
• A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
• Achieving Human Parity on Visual Question Answering
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• Granular Multimodal Attention Networks for Visual Dialog
• Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Transfer Learning in Visual and Relational Reasoning
• Deep Exemplar Networks for VQA and VQG
• Deep Multi-Modal Sets
• Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
• Recurrent Multimodal Interaction for Referring Image Segmentation
• Visual Explanations from Hadamard Product in Multimodal Deep Networks
• Dynamic Attention Networks for Task Oriented Grounding
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Approximated Bilinear Modules for Temporal Modeling
• Proposing Plausible Answers for Open-ended Visual Question Answering
• Adversarial Multimodal Network for Movie Question Answering
• Video SemNet: Memory-Augmented Video Semantic Network
• Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
• Adventurer's Treasure Hunt: A Transparent System for Visually Grounded Compositional Visual Question Answering based on Scene Graphs
• Guessing State Tracking for Visual Dialogue
• Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture
• Parallel Intent and Slot Prediction using MLB Fusion
• Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion
• Pay Attention to Those Sets! Learning Quantification from Images
• Uncertainty based Class Activation Maps for Visual Question Answering
• CS-VQA: Visual Question Answering with Compressively Sensed Images