An Analysis of Categorical Distributional Reinforcement Learning
arXiv:1802.08163
Abstract
Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributional reinforcement learning (CDRL) [Bellemare et al., 2017]. However, the theoretical properties of CDRL algorithms are not yet well understood. In this paper, we introduce a framework to analyse CDRL algorithms, establish the importance of the projected distributional Bellman operator in distributional RL, draw fundamental connections between CDRL and the Cramér distance, and give a proof of convergence for sample-based categorical distributional reinforcement learning algorithms.
References in corpus (2)
Cited by in corpus (13)
- Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
- A Comparative Analysis of Expected and Distributional Reinforcement Learning
- Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
- Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
- Learning to Score Behaviors for Guided Policy Optimization
- Universal Off-Policy Evaluation
- Learning Retrospective Knowledge with Reverse Reinforcement Learning
- Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
- Distributional Reinforcement Learning with Ensembles
- Path-dependent Poisson random measures and stochastic integrals constructed from general point processes
- A Study of Policy Gradient on a Class of Exactly Solvable Models