Learning by Abstraction: The Neural State Machine

Abstract

We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.

Published as a conference paper at NeurIPS 2019 (spotlight)

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Semi-Supervised Classification with Graph Convolutional Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Dynamic Memory Networks for Visual and Textual Question Answering
• Adversarial Examples for Evaluating Reading Comprehension Systems
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Towards Deep Symbolic Reinforcement Learning
• Measuring abstract reasoning in neural networks
• Early Visual Concept Learning with Unsupervised Deep Learning
• The Consciousness Prior
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• SCAN: Learning Hierarchical Compositional Visual Concepts
• Relation-Aware Graph Attention Network for Visual Question Answering
• Compositional Obverter Communication Learning From Raw Visual Input
• Knowledge-Embedded Routing Network for Scene Graph Generation
• Measuring Compositionality in Representation Learning
• Survey of Visual Question Answering: Datasets and Techniques
• Graph R-CNN for Scene Graph Generation
• Answer Them All! Toward Universal Visual Question Answering Models

Cited by in corpus (8)

• Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals
• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• Bridging Knowledge Graphs to Generate Scene Graphs
• Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
• Interpretable Visual Reasoning via Induced Symbolic Space
• Transfer Learning in Visual and Relational Reasoning
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
• Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images