Towards Deep Symbolic Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available. Moreover, they lack the ability to reason on an abstract level, which makes it difficult to implement high-level cognitive functions such as transfer learning, analogical reasoning, and hypothesis-based reasoning. Finally, their operation is largely opaque to humans, rendering them unsuitable for domains in which verifiability is important. In this paper, we propose an end-to-end reinforcement learning architecture comprising a neural back end and a symbolic front end with the potential to overcome each of these shortcomings. As proof-of-concept, we present a preliminary implementation of the architecture and apply it to several variants of a simple video game. We show that the resulting system -- though just a prototype -- learns effectively, and, by acquiring a set of symbolic rules that are easily comprehensible to humans, dramatically outperforms a conventional, fully neural DRL system on a stochastic variant of the game.

References in corpus (8)

• Deep Learning in Neural Networks: An Overview
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Successor Features for Transfer in Reinforcement Learning
• Early Visual Concept Learning with Unsupervised Deep Learning
• Strategic Attentive Writer for Learning Macro-Actions
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
• Tagger: Deep Unsupervised Perceptual Grouping

Cited by in corpus (50)

• A Brief Survey of Deep Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• A simple neural network module for relational reasoning
• Relational Deep Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Compositional Attention Networks for Machine Reasoning
• Acquisition of Chess Knowledge in AlphaZero
• Programmatically Interpretable Reinforcement Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• A Boxology of Design Patterns for Hybrid Learning and Reasoning Systems
• Neurosymbolic Reinforcement Learning and Planning: A Survey
• Conditional Neural Processes
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• An Explicitly Relational Neural Network Architecture
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Learning by Abstraction: The Neural State Machine
• Deliberative Acting, Online Planning and Learning with Hierarchical Operational Models
• Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge
• A Survey of Knowledge-based Sequential Decision Making under Uncertainty
• Towards Symbolic Reinforcement Learning with Common Sense
• Modular Design Patterns for Hybrid Learning and Reasoning Systems: a taxonomy, patterns and use cases
• Analyzing Differentiable Fuzzy Logic Operators
• Faster and Safer Training by Embedding High-Level Knowledge into Deep Reinforcement Learning
• Neuro-Symbolic Constraint Programming for Structured Prediction
• ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning
• Symbolic Network: Generalized Neural Policies for Relational MDPs
• Playing against Nature: causal discovery for decision making under uncertainty
• Injecting Prior Knowledge for Transfer Learning into Reinforcement Learning Algorithms using Logic Tensor Networks
• Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis
• A Survey on Neural-symbolic Learning Systems
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• From Gameplay to Symbolic Reasoning: Learning SAT Solver Heuristics in the Style of Alpha(Go) Zero
• Learning to Play General Video-Games via an Object Embedding Network
• Using Program Induction to Interpret Transition System Dynamics
• Explaining Transition Systems through Program Induction
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• A method for the online construction of the set of states of a Markov Decision Process using Answer Set Programming
• Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions
• SLDR-DL: A Framework for SLD-Resolution with Deep Learning
• Transferable Task Execution from Pixels through Deep Planning Domain Learning
• A Logical Neural Network Structure With More Direct Mapping From Logical Relations
• Formal Verification of End-to-End Learning in Cyber-Physical Systems: Progress and Challenges
• Emergent cooperation through mutual information maximization
• Formal Fields: A Framework to Automate Code Generation Across Domains
• Reinforcement Learning via Reasoning from Demonstration
• Joint Perception and Control as Inference with an Object-based Implementation
• A Novel Neural Network Structure Constructed according to Logical Relations
• Plug and Play, Model-Based Reinforcement Learning