Bayesian Distributional Policy Gradients

Abstract

Distributional Reinforcement Learning (RL) maintains the entire probability distribution of the reward-to-go, i.e. the return, providing more learning signals that account for the uncertainty associated with policy performance, which may be beneficial for trading off exploration and exploitation and policy learning in general. Previous works in distributional RL focused mainly on computing the state-action-return distributions, here we model the state-return distributions. This enables us to translate successful conventional RL algorithms that are based on state values into distributional RL. We formulate the distributional Bellman operation as an inference-based auto-encoding process that minimises Wasserstein metrics between target/model return distributions. The proposed algorithm, BDPG (Bayesian Distributional Policy Gradients), uses adversarial training in joint-contrastive learning to estimate a variational posterior from the returns. Moreover, we can now interpret the return prediction uncertainty as an information gain, which allows to obtain a new curiosity measure that helps BDPG steer exploration actively and efficiently. We demonstrate in a suite of Atari 2600 games and MuJoCo tasks, including well known hard-exploration challenges, how BDPG learns generally faster and with higher asymptotic performance than reference distributional RL algorithms.

References in corpus (9)

• Weight Uncertainty in Neural Networks
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• From optimal transport to generative modeling: the VEGAN cookbook
• UCB Exploration via Q-Ensembles
• Better Exploration with Optimistic Actor-Critic
• Distributional Reinforcement Learning for Efficient Exploration
• Statistics and Samples in Distributional Reinforcement Learning
• Improving Policy Gradient by Exploring Under-appreciated Rewards