A Comparative Analysis of Expected and Distributional Reinforcement Learning

Abstract

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the reasons behind the improvements distributional RL provides. In this paper we begin the investigation into this fundamental question by analyzing the differences in the tabular, linear approximation, and non-linear approximation settings. We prove that in many realizations of the tabular and linear approximation settings, distributional RL behaves exactly the same as expected RL. In cases where the two methods behave differently, distributional RL can in fact hurt performance when it does not induce identical behaviour. We then continue with an empirical analysis comparing distributional and expected RL methods in control settings with non-linear approximators to tease apart where the improvements from distributional RL methods are coming from.

arXiv commentTo appear in the Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence

References in corpus (6)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• An Analysis of Categorical Distributional Reinforcement Learning
• Improving Regression Performance with Distributional Losses

Cited by in corpus (11)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Phasic Policy Gradient
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Policy Evaluation Networks
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mean Embeddings
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms
• A Study of Policy Gradient on a Class of Exactly Solvable Models