Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Abstract

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

arXiv commentNeurIPS 2018 (spotlight). The first two authors contributed equally to this work. Project page: http://nsvqa.csail.mit.edu

References in corpus (7)

• Deep Convolutional Inverse Graphics Network
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• DeepCoder: Learning to Write Programs
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Neuro-Symbolic Program Synthesis
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

Cited by in corpus (63)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Learning to Assemble Neural Module Tree Networks for Visual Grounding
• Discrete and continuous representations and processing in deep learning: Looking forward
• Language-Conditioned Graph Networks for Relational Reasoning
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• On Grounded Planning for Embodied Tasks with Language Models
• GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• In Defense of Grid Features for Visual Question Answering
• Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder
• Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
• Meta Module Network for Compositional Visual Reasoning
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• Why Build an Assistant in Minecraft?
• Differentiable Logic Machines
• Neuro-Symbolic Artificial Intelligence: Current Trends
• Interpretable Visual Question Answering by Reasoning on Dependency Trees
• Compositional Generalization via Neural-Symbolic Stack Machines
• SORNet: Spatial Object-Centric Representations for Sequential Manipulation
• Attention over learned object embeddings enables complex visual reasoning
• Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
• Graph Neural Reasoning May Fail in Certifying Boolean Unsatisfiability
• Relational Programming with Foundation Models
• 2nd Place Solution to the GQA Challenge 2019
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logic
• Learning Task-General Representations with Generative Neuro-Symbolic Modeling
• Gaining Free or Low-Cost Transparency with Interpretable Partial Substitute
• Better Set Representations For Relational Reasoning
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• Unsupervised and interpretable scene discovery with Discrete-Attend-Infer-Repeat
• Neural Logic Reasoning
• A Survey on Neural-symbolic Learning Systems
• Measuring Faithful and Plausible Visual Grounding in VQA
• Visual Superordinate Abstraction for Robust Concept Learning
• NAAQA: A Neural Architecture for Acoustic Question Answering
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination
• Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions
• Interpretable Visual Reasoning via Induced Symbolic Space
• Simulated Mental Imagery for Robotic Task Planning
• Recursive Sketches for Modular Deep Learning
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
• Language-Mediated, Object-Centric Representation Learning
• Playing by the Book: An Interactive Game Approach for Action Graph Extraction from Text
• pix2rule: End-to-end Neuro-symbolic Rule Learning
• Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images
• Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network
• Interpretable Neural Computation for Real-World Compositional Visual Question Answering
• Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering
• Self-Labeling of Fully Mediating Representations by Graph Alignment
• VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks
• Adventurer's Treasure Hunt: A Transparent System for Visually Grounded Compositional Visual Question Answering based on Scene Graphs
• Deep Algorithmic Question Answering: Towards a Compositionally Hybrid AI for Algorithmic Reasoning