Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics

Abstract

The recent adaptation of deep neural network-based methods to reinforcement learning and planning domains has yielded remarkable progress on individual tasks. Nonetheless, progress on task-to-task transfer remains limited. In pursuit of efficient and robust generalization, we introduce the Schema Network, an object-oriented generative physics simulator capable of disentangling multiple causes of events and reasoning backward through causes to achieve goals. The richly structured architecture of the Schema Network can learn the dynamics of an environment directly from data. We compare Schema Networks with Asynchronous Advantage Actor-Critic and Progressive Networks on a suite of Breakout variations, reporting results on training efficiency and zero-shot generalization, consistently demonstrating faster, more robust learning and better transfer. We argue that generalizing from limited data and learning causal relationships are essential abilities on the path toward generally intelligent systems.

References in corpus (2)

• Towards Deep Symbolic Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

Cited by in corpus (50)

• A Brief Survey of Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Deep Learning: A Critical Appraisal
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Relational Deep Reinforcement Learning
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Assessing Generalization in Deep Reinforcement Learning
• Investigating Human Priors for Playing Video Games
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning Plannable Representations with Causal InfoGAN
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Unsupervised Video Object Segmentation for Deep Reinforcement Learning
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Unsupervised State Representation Learning in Atari
• An Explicitly Relational Neural Network Architecture
• A Theory of Relation Learning and Cross-domain Generalization
• Human-like generalization in a machine through predicate learning
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• A Critical Investigation of Deep Reinforcement Learning for Navigation
• ASNets: Deep Learning for Generalised Planning
• Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions
• Graph-Structured Visual Imitation
• Generalized Hindsight for Reinforcement Learning
• Towards Symbolic Reinforcement Learning with Common Sense
• Meta-learning autoencoders for few-shot prediction
• Unsupervised Discovery of 3D Physical Objects from Video
• Learning and Planning with a Semantic Model
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• Deep Abstract Q-Networks
• Grounding Language for Transfer in Deep Reinforcement Learning
• Modularization of End-to-End Learning: Case Study in Arcade Games
• ADARES: Adaptive Resource Management for Virtual Machines
• Cross-View Policy Learning for Street Navigation
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning
• Few-Shot Bayesian Imitation Learning with Logical Program Policies
• Structure Mapping for Transferability of Causal Models
• Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games
• Plug and Play, Model-Based Reinforcement Learning
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• A New Framework for Machine Intelligence: Concepts and Prototype
• Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation
• Computational principles of intelligence: learning and reasoning with neural networks
• Episodic Memory for Learning Subjective-Timescale Models