Sample-based Distributional Policy Gradient

Abstract

Distributional reinforcement learning (DRL) is a recent reinforcement learning framework whose success has been supported by various empirical studies. It relies on the key idea of replacing the expected return with the return distribution, which captures the intrinsic randomness of the long term rewards. Most of the existing literature on DRL focuses on problems with discrete action space and value based methods. In this work, motivated by applications in robotics with continuous action space control settings, we propose sample-based distributional policy gradient (SDPG) algorithm. It models the return distribution using samples via a reparameterization technique widely used in generative modeling and inference. We compare SDPG with the state-of-art policy gradient method in DRL, distributed distributional deterministic policy gradients (D4PG), which has demonstrated state-of-art performance. We apply SDPG and D4PG to multiple OpenAI Gym environments and observe that our algorithm shows better sample efficiency as well as higher reward for most tasks.

References in corpus (5)

• DeepMind Control Suite
• Distributional Reinforcement Learning with Quantile Regression
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• Distributional Multivariate Policy Evaluation and Exploration with the Bellman GAN
• QUOTA: The Quantile Option Architecture for Reinforcement Learning

Cited by in corpus (5)

• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Implicit Distributional Reinforcement Learning
• Adaptive Risk Sensitive Model Predictive Control with Stochastic Search
• GMAC: A Distributional Perspective on Actor-Critic Framework