Exploring Models and Data for Image Question Answering

Abstract

This work aims to address the problem of image-based question-answering (QA) with new models and datasets. In our work, we propose to use neural networks and visual semantic embeddings, without intermediate stages such as object detection and image segmentation, to predict answers to simple questions about images. Our model performs 1.8 times better than the only published results on an existing image QA dataset. We also present a question generation algorithm that converts image descriptions, which are widely available, into QA form. We used this algorithm to produce an order-of-magnitude larger dataset, with more evenly distributed answers. A suite of baseline results on this new dataset are also presented.

12 pages. Conference paper at NIPS 2015

References in corpus (12)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Explain Images with Multimodal Recurrent Neural Networks
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Learning a Recurrent Visual Representation for Image Caption Generation
• Exploring Nearest Neighbor Approaches for Image Captioning
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
• Phrase-based Image Captioning
• Learning to Answer Questions From Image Using Convolutional Neural Network

Cited by in corpus (63)

• Generative Adversarial Text to Image Synthesis
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Mixed Neural Network Approach for Temporal Sleep Stage Classification
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Multimodal Residual Learning for Visual QA
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Representation Learning for Natural Language Processing
• Focal Visual-Text Attention for Visual Question Answering
• Multi-modal Deep Analysis for Multimedia
• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Reasoning Visual Dialogs with Structural and Partial Observations
• EVJVQA Challenge: Multilingual Visual Question Answering
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering
• Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Factor Graph Attention
• Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations
• LEAF-QA: Locate, Encode & Attend for Figure Question Answering
• MCQA: Multimodal Co-attention Based Network for Question Answering
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
• Deep Bayesian Active Learning for Multiple Correct Outputs
• VQA-LOL: Visual Question Answering under the Lens of Logic
• Information Maximizing Visual Question Generation
• A Dataset and Baselines for Visual Question Answering on Art
• Visual Question Answering on 360° Images
• Generating Question Relevant Captions to Aid Visual Question Answering
• Multimodal Learning for Hateful Memes Detection
• Constructing Hierarchical Q&A Datasets for Video Story Understanding
• Multimodal Differential Network for Visual Question Generation
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• Answering Questions about Data Visualizations using Efficient Bimodal Fusion
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• An Entropy Clustering Approach for Assessing Visual Question Difficulty
• CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding
• Privacy-Preserving Federated Learning on Partitioned Attributes
• CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense
• Long Short-Term Memory Spatial Transformer Network
• Deep Reason: A Strong Baseline for Real-World Visual Reasoning
• Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning
• Visual Question Answering as a Multi-Task Problem
• BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
• Learning Question-Guided Video Representation for Multi-Turn Video Question Answering
• Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation
• Granular Multimodal Attention Networks for Visual Dialog
• DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog
• Deep Exemplar Networks for VQA and VQG
• Modeling Dyadic Conversations for Personality Inference
• XtracTree: a Simple and Effective Method for Regulator Validation of Bagging Methods Used in Retail Banking
• Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
• Dynamic Attention Networks for Task Oriented Grounding
• VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks
• GoG: Relation-aware Graph-over-Graph Network for Visual Dialog
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering
• Neural Learning of Online Consumer Credit Risk
• Chat-crowd: A Dialog-based Platform for Visual Layout Composition
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Domain-robust VQA with diverse datasets and methods but no target labels
• Analysis on Image Set Visual Question Answering