Model-Based Reinforcement Learning for Atari

Abstract

Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -- substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.

References in corpus (19)

• Auto-Encoding Variational Bayes
• Playing Atari with Deep Reinforcement Learning
• Neural Discrete Representation Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Recurrent World Models Facilitate Policy Evolution
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Latent Dynamics for Planning from Pixels
• Model-Ensemble Trust-Region Policy Optimization
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Benchmarking Model-Based Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Recurrent Environment Simulators
• When to use parametric models in reinforcement learning?
• Discrete Autoencoders for Sequence Models
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces
• Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Learning Powerful Policies by Using Consistent Dynamics Model

Cited by in corpus (128)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Soft Actor-Critic for Discrete Action Settings
• Dream to Control: Learning Behaviors by Latent Imagination
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• When to Trust Your Model: Model-Based Policy Optimization
• An Optimistic Perspective on Offline Reinforcement Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Contrastive Learning of Structured World Models
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Reinforcement Learning through Active Inference
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Will we ever have Conscious Machines?
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Trust-Region Method with Deep Reinforcement Learning in Analog Design Space Exploration
• Mastering Atari with Discrete World Models
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• Multi-Task Reinforcement Learning with Context-based Representations
• High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• Explanation-Aware Experience Replay in Rule-Dense Environments
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Fast Task Inference with Variational Intrinsic Successor Features
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Causally Correct Partial Models for Reinforcement Learning
• Combining Q-Learning and Search with Amortized Value Estimates
• On the model-based stochastic value gradient for continuous reinforcement learning
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Modern Deep Reinforcement Learning Algorithms
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Scaling Autoregressive Video Models
• Mathematical Reasoning in Latent Space
• Model-based Reinforcement Learning for Predictions and Control for Limit Order Books
• Reinforcement Learning with Dual-Observation for General Video Game Playing
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Behavior From the Void: Unsupervised Active Pre-Training
• Skip Training for Multi-Agent Reinforcement Learning Controller for Industrial Wave Energy Converters
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Learning Human Objectives by Evaluating Hypothetical Behavior
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Is Plug-in Solver Sample-Efficient for Feature-based Reinforcement Learning?
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Reinforcement Learning for Load-balanced Parallel Particle Tracing
• Entanglement engineering of optomechanical systems by reinforcement learning
• A Probabilistic Simulator of Spatial Demand for Product Allocation
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Uncertainty-sensitive Learning and Planning with Ensembles
• Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Minimax Sample Complexity for Turn-based Stochastic Game
• Learning World Transition Model for Socially Aware Robot Navigation
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Learning "What-if" Explanations for Sequential Decision-Making
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Continual Learning Using World Models for Pseudo-Rehearsal
• Machine Learning for Mechanical Ventilation Control
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Sample-efficient reinforcement learning using deep Gaussian processes
• Active Perception and Representation for Robotic Manipulation
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• Predictive Coding for Locally-Linear Control
• Off-Policy Actor-Critic with Shared Experience Replay
• Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Adversarial recovery of agent rewards from latent spaces of the limit order book
• Learning a Generative Transition Model for Uncertainty-Aware Robotic Manipulation
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue Task
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Safely Learning Dynamical Systems from Short Trajectories
• Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates
• Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver
• Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
• Disentangling Controllable Object through Video Prediction Improves Visual Reinforcement Learning
• Frequency-based Search-control in Dyna
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• The Principle of Unchanged Optimality in Reinforcement Learning Generalization
• Pretrained Encoders are All You Need
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
• Supervised Off-Policy Ranking
• Sim-Env: Decoupling OpenAI Gym Environments from Simulation Models
• Improving Experience Replay through Modeling of Similar Transitions' Sets
• Learning Representations for Pixel-based Control: What Matters and Why?
• Deep Gaussian Covariance Network with Trajectory Sampling for Data-Efficient Policy Search
• Recomposing the Reinforcement Learning Building Blocks with Hypernetworks
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
• Evaluating the Apperception Engine
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation
• Neuro-evolutionary Frameworks for Generalized Learning Agents
• On-Policy Model Errors in Reinforcement Learning
• Planning with Expectation Models for Control
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• Reusing Convolutional Activations from Frame to Frame to Speed up Training and Inference
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution
• On The Transferability of Deep-Q Networks
• The Efficiency Misnomer
• Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning
• Episodic Memory for Learning Subjective-Timescale Models
• Zero Shot Learning on Simulated Robots
• Low-Variance Policy Gradient Estimation with World Models
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
• Model-Augmented Q-learning
• Enhancing Reinforcement Learning Through Guided Search
• Discriminator Augmented Model-Based Reinforcement Learning