A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Abstract

We propose a method for automatically answering questions about images by bringing together recent advances from natural language processing and computer vision. We combine discrete reasoning with uncertain predictions by a multi-world approach that represents uncertainty about the perceived world in a bayesian framework. Our approach can handle human questions of high complexity about realistic scenes and replies with range of answer like counts, object classes, instances and lists of them. The system is directly trained from question-answer pairs. We establish a first benchmark for this task that can be seen as a modern attempt at a visual turing test.

Published in NIPS 2014

Cited by in corpus (29)

• Vision-to-Language Tasks Based on Attributes and Attention Mechanism
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
• Reasoning Visual Dialogs with Structural and Partial Observations
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
• Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions
• Deep Bayesian Active Learning for Multiple Correct Outputs
• Leveraging Medical Visual Question Answering with Supporting Facts
• A Dataset and Baselines for Visual Question Answering on Art
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Answer Them All! Toward Universal Visual Question Answering Models
• Visual Question Answering on 360° Images
• Weakly supervised cross-domain alignment with optimal transport
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
• Deep Reason: A Strong Baseline for Real-World Visual Reasoning
• Visual Question Answering as a Multi-Task Problem
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• Deep Exemplar Networks for VQA and VQG
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Granular Multimodal Attention Networks for Visual Dialog
• Selective Replay Enhances Learning in Online Continual Analogical Reasoning
• Understanding in Artificial Intelligence
• Analysis on Image Set Visual Question Answering
• Dynamic Attention Networks for Task Oriented Grounding