Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
arXiv:2007.14535
Abstract
In the present paper, we propose a decoder-free extension of Dreamer, a leading model-based reinforcement learning (MBRL) method from pixels. Dreamer is a sample- and cost-efficient solution to robot learning, as it is used to train latent state-space models based on a variational autoencoder and to conduct policy optimization by latent trajectory imagination. However, this autoencoding based approach often causes object vanishing, in which the autoencoder fails to perceives key objects for solving control tasks, and thus significantly limiting Dreamer's potential. This work aims to relieve this Dreamer's bottleneck and enhance its performance by means of removing the decoder. For this purpose, we firstly derive a likelihood-free and InfoMax objective of contrastive learning from the evidence lower bound of Dreamer. Secondly, we incorporate two components, (i) independent linear dynamics and (ii) the random crop data augmentation, to the learning scheme so as to improve the training performance. In comparison to Dreamer and other recent model-free reinforcement learning methods, our newly devised Dreamer with InfoMax and without generative decoder (Dreaming) achieves the best scores on 5 difficult simulated robotics tasks, in which Dreamer suffers from object vanishing.
arXiv commentAccepted to ICRA2021. Camera ready version
References in corpus (21)
- Auto-Encoding Variational Bayes
- A Simple Framework for Contrastive Learning of Visual Representations
- Representation Learning with Contrastive Predictive Coding
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- DeepMind Control Suite
- Recurrent World Models Facilitate Policy Evolution
- Learning Latent Dynamics for Planning from Pixels
- On Mutual Information Maximization for Representation Learning
- Benchmarking Model-Based Reinforcement Learning
- On Variational Bounds of Mutual Information
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
- Planning to Explore via Self-Supervised World Models
- SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
- Variational Inference MPC for Bayesian Model-based Reinforcement Learning
- Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
- Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
- Curious iLQR: Resolving Uncertainty in Model-based RL
- Asynchronous Methods for Model-Based Reinforcement Learning