Simple Baseline for Visual Question Answering

Abstract

We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to predict the answer. When evaluated on the challenging VQA dataset [2], it shows comparable performance to many recent approaches using recurrent neural networks. To explore the strength and weakness of the trained model, we also provide an interactive web demo and open-source code. .

One comparison method's scores are put into the correct column, and a new experiment of generating attention map is added

References in corpus (7)

• Going Deeper with Convolutions
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Learning Deep Features for Discriminative Localization
• Exploring Nearest Neighbor Approaches for Image Captioning
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Compositional Memory for Visual Question Answering

Cited by in corpus (80)

• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Dual Encoding for Video Retrieval by Text
• Learning Deep Features for Discriminative Localization
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• A Focused Dynamic Attention Model for Visual Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Deep Modular Co-Attention Networks for Visual Question Answering
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Person Search with Natural Language Description
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Zero-Shot Visual Question Answering
• Dual Attention Networks for Multimodal Reasoning and Matching
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Visual Question Answering: A Survey of Methods and Datasets
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• A Multimodal Memes Classification: A Survey and Open Research Issues
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Revisiting Visual Question Answering Baselines
• Scene Graph Generation from Objects, Phrases and Region Captions
• Visual Question Generation as Dual Task of Visual Question Answering
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• Question-Guided Hybrid Convolution for Visual Question Answering
• Survey of Visual Question Answering: Datasets and Techniques
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Visual Question Answering with Memory-Augmented Networks
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Compact Trilinear Interaction for Visual Question Answering
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Ways of Conditioning Generative Adversarial Networks
• Reciprocal Attention Fusion for Visual Question Answering
• Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
• Deep Bayesian Active Learning for Multiple Correct Outputs
• Object-Centric Diagnosis of Visual Reasoning
• MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering
• Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
• MarioQA: Answering Questions by Watching Gameplay Videos
• Dual Encoding for Zero-Example Video Retrieval
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Figure Captioning with Reasoning and Sequence-Level Training
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Visual Question Answering with Prior Class Semantics
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
• Experimental Standards for Deep Learning in Natural Language Processing Research
• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• A Better Loss for Visual-Textual Grounding
• Question Guided Modular Routing Networks for Visual Question Answering
• Trying Bilinear Pooling in Video-QA
• Transfer Learning in Visual and Relational Reasoning
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Leveraging Visual Question Answering for Image-Caption Ranking
• Text classification with pixel embedding
• Will Multi-modal Data Improves Few-shot Learning?
• Recurrent and Contextual Models for Visual Question Answering
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Triple Attention Network architecture for MovieQA
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Proposing Plausible Answers for Open-ended Visual Question Answering
• Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
• A Free Lunch in Generating Datasets: Building a VQG and VQA System with Attention and Humans in the Loop
• Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
• Pay Attention to Those Sets! Learning Quantification from Images
• A Revised Generative Evaluation of Visual Dialogue
• After All, Only The Last Neuron Matters: Comparing Multi-modal Fusion Functions for Scene Graph Generation