Hadamard Product for Low-rank Bilinear Pooling

Abstract

Bilinear models provide rich representations compared with linear models. They have been applied in various visual tasks, such as object recognition, segmentation, and visual question-answering, to get state-of-the-art performances taking advantage of the expanded representations. However, bilinear representations tend to be high-dimensional, limiting the applicability to computationally complex tasks. We propose low-rank bilinear pooling using Hadamard product for an efficient attention mechanism of multimodal learning. We show that our model outperforms compact bilinear pooling in visual question-answering tasks with the state-of-the-art results on the VQA dataset, having a better parsimonious property.

13 pages, 1 figure, & appendix. ICLR 2017 accepted

Cited by in corpus (34)

• Deep Modular Co-Attention Networks for Visual Question Answering
• DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Visual Question Generation as Dual Task of Visual Question Answering
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• Local-Global Video-Text Interactions for Temporal Grounding
• Structured Attentions for Visual Question Answering
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Multi-modal Summarization for Video-containing Documents
• Position Focused Attention Network for Image-Text Matching
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• Spatial Dual-Modality Graph Reasoning for Key Information Extraction
• Compact Trilinear Interaction for Visual Question Answering
• Progressive Attention Memory Network for Movie Story Question Answering
• Question-Conditioned Counterfactual Image Generation for VQA
• Leveraging Medical Visual Question Answering with Supporting Facts
• Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network
• Deep Claim: Payer Response Prediction from Claims Data with Deep Learning
• Answer Them All! Toward Universal Visual Question Answering Models
• Visual Question Answering on 360° Images
• Multimodal Learning for Hateful Memes Detection
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Overcoming Statistical Shortcuts for Open-ended Visual Counting
• A Joint Network for Grasp Detection Conditioned on Natural Language Commands
• Trying Bilinear Pooling in Video-QA
• Deep Reason: A Strong Baseline for Real-World Visual Reasoning
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Neuron Interaction Based Representation Composition for Neural Machine Translation
• The Resale Price Prediction of Secondhand Jewelry Items Using a Multi-modal Deep Model with Iterative Co-Attention
• Deep Exemplar Networks for VQA and VQG
• Enforcing Reasoning in Visual Commonsense Reasoning
• Approximated Bilinear Modules for Temporal Modeling