Low-Variance Policy Gradient Estimation with World Models

Abstract

In this paper, we propose World Model Policy Gradient (WMPG), an approach to reduce the variance of policy gradient estimates using learned world models (WM's). In WMPG, a WM is trained online and used to imagine trajectories. The imagined trajectories are used in two ways. Firstly, to calculate a without-replacement estimator of the policy gradient. Secondly, the return of the imagined trajectories is used as an informed baseline. We compare the proposed approach with AC and MAC on a set of environments of increasing complexity (CartPole, LunarLander and Pong) and find that WMPG has better sample efficiency. Based on these results, we conclude that WMPG can yield increased sample efficiency in cases where a robust latent representation of the environment can be learned.

References in corpus (12)

• Auto-Encoding Variational Bayes
• Trust Region Policy Optimization
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Learning Latent Dynamics for Planning from Pixels
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Planning to Explore via Self-Supervised World Models
• Incremental Sampling Without Replacement for Sequence Models
• Learning Discrete State Abstractions With Deep Variational Inference