Mastering Atari with Discrete World Models

Abstract

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.

arXiv commentPublished at ICLR 2021. Website: https://danijar.com/dreamerv2

References in corpus (35)

• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• Model-Ensemble Trust-Region Policy Optimization
• MOPO: Model-based Offline Policy Optimization
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Benchmarking Model-Based Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Deep Kalman Filters
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Recurrent Environment Simulators
• Self-Supervised Visual Planning with Temporal Skip Connections
• Universal Planning Networks
• Exploring Model-based Planning with Policy Networks
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Generative Temporal Models with Memory
• Model-Based Planning with Discrete and Continuous Actions
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field

Cited by in corpus (4)

• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• On the Sensory Commutativity of Action Sequences for Embodied Agents
• Learning State Representations from Random Deep Action-conditional Predictions