Dream to Control: Learning Behaviors by Latent Imagination

Abstract

Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

arXiv comment9 pages, 12 figures

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Representation Learning with Contrastive Predictive Coding
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Deep Variational Information Bottleneck
• Learning Continuous Control Policies by Stochastic Value Gradients
• Learning Latent Dynamics for Planning from Pixels
• Distributed Distributional Deterministic Policy Gradients
• Model-Ensemble Trust-Region Policy Optimization
• Benchmarking Model-Based Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Deep Kalman Filters
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Self-Supervised Visual Planning with Temporal Skip Connections
• Universal Planning Networks
• Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
• Exploring Model-based Planning with Policy Networks
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Formal Limitations on the Measurement of Mutual Information
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Neural Predictive Belief Representations
• Robust Locally-Linear Controllable Embedding
• Model-Based Planning with Discrete and Continuous Actions
• Probabilistic Recurrent State-Space Models
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Cited by in corpus (54)

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Reinforcement Learning through Active Inference
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Reinforcement Learning with Prototypical Representations
• Simple and Effective VAE Training with Calibrated Decoders
• A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• Action and Perception as Divergence Minimization
• Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
• Deep Reinforcement and InfoMax Learning
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Model-Based Offline Planning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Model-Based Visual Planning with Self-Supervised Functional Distances
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Learning and Planning in Complex Action Spaces
• Intervention Design for Effective Sim2Real Transfer
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model
• Contrastive Variational Reinforcement Learning for Complex Observations
• Local Search for Policy Iteration in Continuous Control
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Continual Model-Based Reinforcement Learning with Hypernetworks
• The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Centralized Model and Exploration Policy for Multi-Agent RL
• Reinforced Imitation Learning by Free Energy Principle
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Sampling Network Guided Cross-Entropy Method for Unsupervised Point Cloud Registration
• Learning to Correspond Dynamical Systems
• Curiosity creates Diversity in Policy Search
• Information-Theoretic Odometry Learning
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Learning Space Partitions for Path Planning
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Reinforcement Learning with Latent Flow
• Assisted Perception: Optimizing Observations to Communicate State
• Pretrained Encoders are All You Need
• Dream and Search to Control: Latent Space Planning for Continuous Control
• Extracting Latent State Representations with Linear Dynamics from Rich Observations
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Cautious Actor-Critic
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Visual Perspective Taking for Opponent Behavior Modeling
• Learn Proportional Derivative Controllable Latent Space from Pixels
• Self-Consistent Models and Values
• On The Transferability of Deep-Q Networks
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training