Implicit Distributional Reinforcement Learning
arXiv:2007.06159
Abstract
To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor-critic (IDAC) that consists of a distributional critic, built on two deep generator networks (DGNs), and a semi-implicit actor (SIA), powered by a flexible policy distribution. We adopt a distributional perspective on the discounted cumulative return and model it with a state-action-dependent implicit distribution, which is approximated by the DGNs that take state-action pairs and random noises as their input. Moreover, we use the SIA to provide a semi-implicit policy distribution, which mixes the policy parameters with a reparameterizable distribution that is not constrained by an analytic density function. In this way, the policy's marginal distribution is implicit, providing the potential to model complex properties such as covariance structure and skewness, but its parameter and entropy can still be estimated. We incorporate these features with an off-policy algorithm framework to solve problems with continuous action space and compare IDAC with state-of-the-art algorithms on representative OpenAI Gym environments. We observe that IDAC outperforms these baselines in most tasks. Python code is provided.
arXiv commentNeurIPS 2020
References in corpus (15)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic Algorithms and Applications
- Density estimation using Real NVP
- Reinforcement Learning with Deep Energy-Based Policies
- Distributed Distributional Deterministic Policy Gradients
- The Cramer Distance as a Solution to Biased Wasserstein Gradients
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Distributional Reinforcement Learning with Quantile Regression
- Approximate Bayesian computation with the Wasserstein distance
- Distributional Policy Optimization: An Alternative Approach for Continuous Control
- Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
- Sample-based Distributional Policy Gradient
- Implicit Policy for Reinforcement Learning