Fully Parameterized Quantile Function for Distributional Reinforcement Learning

Abstract

Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.

arXiv commentNeurIPS 2019. Code at https://github.com/microsoft/FQF

Cited by in corpus (13)

• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Probabilistic Time Series Forecasting with Implicit Quantile Networks
• Cautious Reinforcement Learning via Distributional Risk in the Dual Domain
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
• Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
• Safe Distributional Reinforcement Learning
• Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• GMAC: A Distributional Perspective on Actor-Critic Framework