Learning Latent Dynamics for Planning from Pixels

Abstract

Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher than strong model-free algorithms.

arXiv comment20 pages, 12 figures, 1 table

Cited by in corpus (188)

• On the Opportunities and Risks of Foundation Models
• Model-Based Reinforcement Learning for Atari
• Challenges of Real-World Reinforcement Learning
• Reinforcement Learning with Augmented Data
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Recurrent Independent Mechanisms
• Contrastive Learning of Structured World Models
• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
• Decoupling Representation Learning from Reinforcement Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Planning to Explore via Self-Supervised World Models
• Learning Compositional Koopman Operators for Model-Based Control
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Causal Discovery in Physical Systems from Videos
• PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks
• Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Simple and Effective VAE Training with Calibrated Decoders
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• FitVid: Overfitting in Pixel-Level Video Prediction
• Data Efficient Reinforcement Learning for Legged Robots
• Hamiltonian Generative Networks
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity
• 3D Neural Scene Representations for Visuomotor Control
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Deep Reinforcement and InfoMax Learning
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• Mastering Atari with Discrete World Models
• Predictive Information Accelerates Learning in RL
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Visual Grounding of Learned Physical Models
• World Model as a Graph: Learning Latent Landmarks for Planning
• High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• Model-Based Offline Planning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Unsupervised Learning of Object Structure and Dynamics from Videos
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Solving PDE-constrained Control Problems Using Operator Learning
• Offline Reinforcement Learning from Images with Latent Space Models
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• MBRL-Lib: A Modular Library for Model-based Reinforcement Learning
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
• Optimistic Exploration even with a Pessimistic Initialisation
• Recurrent Off-policy Baselines for Memory-based Continuous Control
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Learning Latent Representations to Influence Multi-Agent Interaction
• Clockwork Variational Autoencoders
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Latent Representation in Human-Robot Interaction with Explicit Consideration of Periodic Dynamics
• Model-free Representation Learning and Exploration in Low-rank MDPs
• Reset-Free Lifelong Learning with Skill-Space Planning
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Exploration and preference satisfaction trade-off in reward-free learning
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Procedure Planning in Instructional Videos
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Representation Learning for Event-based Visuomotor Policies
• Generative Neurosymbolic Machines
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Learning Human Objectives by Evaluating Hypothetical Behavior
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
• Ready Policy One: World Building Through Active Learning
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• An Empirical Investigation of Representation Learning for Imitation
• Forward Prediction for Physical Reasoning
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Local Search for Policy Iteration in Continuous Control
• Object-centric Forward Modeling for Model Predictive Control
• Deep Affordance Foresight: Planning Through What Can Be Done in the Future
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Continual Model-Based Reinforcement Learning with Hypernetworks
• XLVIN: eXecuted Latent Value Iteration Nets
• DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Attention-Privileged Reinforcement Learning
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Selective Dyna-style Planning Under Limited Model Capacity
• Action-Based Representation Learning for Autonomous Driving
• Visual Adversarial Imitation Learning using Variational Models
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Inference of Affordances and Active Motor Control in Simulated Agents
• On learning latent dynamics of the AUG plasma state
• Learning World Transition Model for Socially Aware Robot Navigation
• Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature
• SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics
• Implicit Generative Modeling for Efficient Exploration
• Few Shot System Identification for Reinforcement Learning
• Learning Dynamics Models for Model Predictive Agents
• Machine Learning for Mechanical Ventilation Control
• Iterative Amortized Policy Optimization
• Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
• Dynamic Future Net: Diversified Human Motion Generation
• Sampling Network Guided Cross-Entropy Method for Unsupervised Point Cloud Registration
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Regularizing Model-Based Planning with Energy-Based Models
• Feedback from Pixels: Output Regulation via Learning-Based Scene View Synthesis
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Planning from Pixels using Inverse Dynamics Models
• Model-Based Episodic Memory Induces Dynamic Hybrid Controls
• Influence-Augmented Online Planning for Complex Environments
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• Learning Latent State Spaces for Planning through Reward Prediction
• Block Contextual MDPs for Continual Learning
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Learning Space Partitions for Path Planning
• LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Deep Learning Alternative to Explicit Model Predictive Control for Unknown Nonlinear Systems
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• Planning with Learned Dynamic Model for Unsupervised Point Cloud Registration
• MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Reinforcement Learning with Latent Flow
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
• Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning
• Relational State-Space Model for Stochastic Multi-Object Systems
• CostNet: An End-to-End Framework for Goal-Directed Reinforcement Learning
• Disentangling Controllable Object through Video Prediction Improves Visual Reinforcement Learning
• Making Curiosity Explicit in Vision-based RL
• Pre-training of Deep RL Agents for Improved Learning under Domain Randomization
• Uncertainty Estimation and Calibration with Finite-State Probabilistic RNNs
• Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning
• Learning the Linear Quadratic Regulator from Nonlinear Observations
• Towards Robust Bisimulation Metric Learning
• Data-Driven Reinforcement Learning for Virtual Character Animation Control
• Learning Representations for Pixel-based Control: What Matters and Why?
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control
• Robo-PlaNet: Learning to Poke in a Day
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
• Dream and Search to Control: Latent Space Planning for Continuous Control
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• Swoosh! Rattle! Thump! -- Actions that Sound
• Planning from Images with Deep Latent Gaussian Process Dynamics
• Visual Reaction: Learning to Play Catch with Your Drone
• Discriminator Augmented Model-Based Reinforcement Learning
• Learning of feature points without additional supervision improves reinforcement learning from images
• On The Transferability of Deep-Q Networks
• Learning What To Do by Simulating the Past
• Neural Algorithmic Reasoners are Implicit Planners
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution
• Learn Proportional Derivative Controllable Latent Space from Pixels
• Maximum Entropy Model-based Reinforcement Learning
• Low-Variance Policy Gradient Estimation with World Models
• Towards Learning Controllable Representations of Physical Systems
• Differentiable Robust LQR Layers
• Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
• Contrastive Active Inference
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions
• Channel Decomposition into Painting Actions
• Policy Search using Dynamic Mirror Descent MPC for Model Free Off Policy RL
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Visual Perspective Taking for Opponent Behavior Modeling