A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Abstract

We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.

arXiv commentPublished in NIPS 2014

References in corpus (1)

• Verb Semantics and Lexical Selection

Cited by in corpus (54)

• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Exploring Models and Data for Image Question Answering
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Generation and Comprehension of Unambiguous Object Descriptions
• PathVQA: 30000+ Questions for Medical Visual Question Answering
• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Visual7W: Grounded Question Answering in Images
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
• Reasoning Visual Dialogs with Structural and Partial Observations
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
• Factor Graph Attention
• On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
• LEAF-QA: Locate, Encode & Attend for Figure Question Answering
• Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
• VQA-LOL: Visual Question Answering under the Lens of Logic
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• Deep Bayesian Active Learning for Multiple Correct Outputs
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• Leveraging Medical Visual Question Answering with Supporting Facts
• Estimating semantic structure for the VQA answer space
• A Dataset and Baselines for Visual Question Answering on Art
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Answer Them All! Toward Universal Visual Question Answering Models
• Visual Question Answering on 360° Images
• Weakly supervised cross-domain alignment with optimal transport
• Answering Questions about Data Visualizations using Efficient Bimodal Fusion
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense
• A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
• Deep Reason: A Strong Baseline for Real-World Visual Reasoning
• Visual Question Answering as a Multi-Task Problem
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• Grounded Textual Entailment
• Deep Exemplar Networks for VQA and VQG
• Selective Replay Enhances Learning in Online Continual Analogical Reasoning
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Granular Multimodal Attention Networks for Visual Dialog
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• $C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
• Semi-supervised Visual Feature Integration for Pre-trained Language Models
• Analysis on Image Set Visual Question Answering
• Dynamic Attention Networks for Task Oriented Grounding
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Spatial Attention as an Interface for Image Captioning Models
• Understanding in Artificial Intelligence