Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Abstract

Reinforcement learning (RL) studies how an agent comes to achieve reward in an environment through interactions over time. Recent advances in machine RL have surpassed human expertise at the world's oldest board games and many classic video games, but they require vast quantities of experience to learn successfully -- none of today's algorithms account for the human ability to learn so many different tasks, so quickly. Here we propose a new approach to this challenge based on a particularly strong form of model-based RL which we call Theory-Based Reinforcement Learning, because it uses human-like intuitive theories -- rich, abstract, causal models of physical objects, intentional agents, and their interactions -- to explore and model an environment, and plan effectively to achieve task goals. We instantiate the approach in a video game playing agent called EMPA (the Exploring, Modeling, and Planning Agent), which performs Bayesian inference to learn probabilistic generative models expressed as programs for a game-engine simulator, and runs internal simulations over these models to support efficient object-based, relational exploration and heuristic planning. EMPA closely matches human learning efficiency on a suite of 90 challenging Atari-style video games, learning new games in just minutes of game play and generalizing robustly to new game situations and new levels. The model also captures fine-grained structure in people's exploration trajectories and learning dynamics. Its design and behavior suggest a way forward for building more general human-like AI systems.

References in corpus (21)

• Prioritized Experience Replay
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Model-Based Reinforcement Learning for Atari
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Agent57: Outperforming the Atari Human Benchmark
• Bayesian Reinforcement Learning: A Survey
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Learning Symbolic Models of Stochastic Domains
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Relational Deep Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Towards Deep Symbolic Reinforcement Learning
• Recurrent Environment Simulators
• Learning Visual Predictive Models of Physics for Playing Billiards
• Causal Reasoning from Meta-reinforcement Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games
• Learning sparse relational transition models