Bilinear Attention Networks

Abstract

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

arXiv commentAccepted by NIPS 2018; Figure 1 was updated

References in corpus (6)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Learning to Count Objects in Natural Images for Visual Question Answering
• Dual Attention Networks for Multimodal Reasoning and Matching
• Interpretable Counting for Visual Question Answering

Cited by in corpus (28)

• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
• Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Visual Entailment Task for Visually-Grounded Language Learning
• Image-Question-Answer Synergistic Network for Visual Dialog
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• Zero-Shot Learning from scratch (ZFS): leveraging local compositional representations
• MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
• TallyQA: Answering Complex Counting Questions
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• Contextual Grounding of Natural Language Entities in Images
• Overcoming Language Priors with Self-supervised Learning for Visual Question Answering
• Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• A New Training Framework for Deep Neural Network
• Interpretable Multimodal Learning for Intelligent Regulation in Online Payment Systems
• Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
• Unsupervised Keyword Extraction for Full-sentence VQA
• Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering
• Uncertainty based Class Activation Maps for Visual Question Answering
• Learning to Represent and Predict Sets with Deep Neural Networks