The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Abstract

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

arXiv commentICLR 2019 (Oral). Project page: http://nscl.csail.mit.edu/

Cited by in corpus (49)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• A Survey of Knowledge-Enhanced Text Generation
• Abstraction and Analogy-Making in Artificial Intelligence
• Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning
• DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering
• On the Binding Problem in Artificial Neural Networks
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• A Comprehensive Survey of Machine Learning Applied to Radar Signal Processing
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• CoPhy: Counterfactual Learning of Physical Dynamics
• Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning
• ProTo: Program-Guided Transformer for Program-Guided Tasks
• Relational Reasoning using Prior Knowledge for Visual Captioning
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Meta Module Network for Compositional Visual Reasoning
• Learning from Lexical Perturbations for Consistent Visual Question Answering
• VQA-LOL: Visual Question Answering under the Lens of Logic
• Explainable Artificial Intelligence (XAI) for 6G: Improving Trust between Human and Machine
• Compositional Generalization via Neural-Symbolic Stack Machines
• Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
• The Sigma-max System Induced from Randomness & Fuzziness and its Application in Time Series Prediction
• Learning Task-General Representations with Generative Neuro-Symbolic Modeling
• A Benchmark and Baseline for Language-Driven Image Editing
• Supervising the Transfer of Reasoning Patterns in VQA
• From Shallow to Deep Interactions Between Knowledge Representation, Reasoning and Machine Learning (Kay R. Amel group)
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• Robot Action Selection Learning via Layered Dimension Informed Program Synthesis
• A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
• Visual Superordinate Abstraction for Robust Concept Learning
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• Multi-Granularity Modularized Network for Abstract Visual Reasoning
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Linguistic Structures as Weak Supervision for Visual Scene Graph Generation
• Interpretable Visual Reasoning via Induced Symbolic Space
• Neural Abstract Reasoner
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• PHYRE: A New Benchmark for Physical Reasoning
• Neuro-Symbolic VQA: A review from the perspective of AGI desiderata
• Web Question Answering with Neurosymbolic Program Synthesis
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Extending Answer Set Programs with Neural Networks
• Learning with Holographic Reduced Representations
• Learning by Planning: Language-Guided Global Image Editing
• Exploring End-to-End Differentiable Natural Logic Modeling
• ACRE: Abstract Causal REasoning Beyond Covariation
• Domain-robust VQA with diverse datasets and methods but no target labels
• Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network