Hadamard Product for Low-rank Bilinear Pooling

Abstract

Bilinear models provide rich representations compared with linear models. They have been applied in various visual tasks, such as object recognition, segmentation, and visual question-answering, to get state-of-the-art performances taking advantage of the expanded representations. However, bilinear representations tend to be high-dimensional, limiting the applicability to computationally complex tasks. We propose low-rank bilinear pooling using Hadamard product for an efficient attention mechanism of multimodal learning. We show that our model outperforms compact bilinear pooling in visual question-answering tasks with the state-of-the-art results on the VQA dataset, having a better parsimonious property.

arXiv comment13 pages, 1 figure, & appendix. ICLR 2017 accepted

Cited by in corpus (64)

• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
• Low-Rank Pairwise Alignment Bilinear Network For Few-Shot Fine-Grained Image Classification
• MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
• Deep Modular Co-Attention Networks for Visual Question Answering
• DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
• TOAN: Target-Oriented Alignment Network for Fine-Grained Image Categorization with Few Labeled Samples
• Relation-Aware Graph Attention Network for Visual Question Answering
• Attention on Attention for Image Captioning
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• Driving Behavior Explanation with Multi-level Fusion
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• Visual Question Generation as Dual Task of Visual Question Answering
• Local-Global Video-Text Interactions for Temporal Grounding
• Structured Attentions for Visual Question Answering
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• Simulating Action Dynamics with Neural Process Networks
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Combination of Deep Speaker Embeddings for Diarisation
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Multi-modal Summarization for Video-containing Documents
• Position Focused Attention Network for Image-Text Matching
• Spatial Dual-Modality Graph Reasoning for Key Information Extraction
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• Compact Trilinear Interaction for Visual Question Answering
• Factor Graph Attention
• Progressive Attention Memory Network for Movie Story Question Answering
• Question-Conditioned Counterfactual Image Generation for VQA
• Meta Module Network for Compositional Visual Reasoning
• X-Linear Attention Networks for Image Captioning
• Deep Multimodal Neural Architecture Search
• Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Leveraging Medical Visual Question Answering with Supporting Facts
• Interpretable Visual Question Answering by Reasoning on Dependency Trees
• Deep Claim: Payer Response Prediction from Claims Data with Deep Learning
• Visual Question Answering on 360° Images
• Answer Them All! Toward Universal Visual Question Answering Models
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• Multimodal Learning for Hateful Memes Detection
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• A Joint Network for Grasp Detection Conditioned on Natural Language Commands
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• Frontal Low-rank Random Tensors for Fine-grained Action Segmentation
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Trying Bilinear Pooling in Video-QA
• Deep Reason: A Strong Baseline for Real-World Visual Reasoning
• Low Rank Factorization for Compact Multi-Head Self-Attention
• Neuron Interaction Based Representation Composition for Neural Machine Translation
• Deep Exemplar Networks for VQA and VQG
• The Resale Price Prediction of Secondhand Jewelry Items Using a Multi-modal Deep Model with Iterative Co-Attention
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Enforcing Reasoning in Visual Commonsense Reasoning
• Phrase Grounding by Soft-Label Chain Conditional Random Field
• REXUP: I REason, I EXtract, I UPdate with Structured Compositional Reasoning for Visual Question Answering
• A Unifying Framework of Bilinear LSTMs
• Approximated Bilinear Modules for Temporal Modeling
• Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition