Distributional Reinforcement Learning with Quantile Regression

Abstract

In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average over this randomness to estimate the value function. In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean. That is, we examine methods of learning the value distribution instead of the value function. We give results that close a number of gaps between the theoretical and algorithmic results given by Bellemare, Dabney, and Munos (2017). First, we extend existing results to the approximate distribution setting. Second, we present a novel distributional reinforcement learning algorithm consistent with our theoretical formulation. Finally, we evaluate this new algorithm on the Atari 2600 games, observing that it significantly outperforms many of the recent improvements on DQN, including the related distributional algorithm C51.

Cited by in corpus (84)

• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• A Survey of Deep Reinforcement Learning in Video Games
• Offline Reinforcement Learning with Implicit Q-Learning
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Discovering Reinforcement Learning Algorithms
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Self-supervised transfer learning of physiological representations from free-living wearable data
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Mastering Atari with Discrete World Models
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Value Penalized Q-Learning for Recommender Systems
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Millimeter Wave Communications with an Intelligent Reflector: Performance Optimization and Distributional Reinforcement Learning
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• Modern Deep Reinforcement Learning Algorithms
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• Off-Policy Risk Assessment in Contextual Bandits
• STEP: Stochastic Traversability Evaluation and Planning for Risk-Aware Off-road Navigation
• An Open-Source Framework for Adaptive Traffic Signal Control
• Offline Reinforcement Learning with Value-based Episodic Memory
• General non-linear Bellman equations
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Adapting Behaviour for Learning Progress
• Sample-based Distributional Policy Gradient
• Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Implicit Distributional Reinforcement Learning
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
• Quantile Markov Decision Process
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
• Learning Risk-aware Costmaps for Traversability in Challenging Environments
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Sample Efficient Ensemble Learning with Catalyst.RL
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
• Learning Policies through Quantile Regression
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• Stochastically Dominant Distributional Reinforcement Learning
• Predictive Uncertainty through Quantization
• Deep Reinforcement Learning with Weighted Q-Learning
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• Adapting to Reward Progressivity via Spectral Reinforcement Learning
• Unsupervised Learning of Solutions to Differential Equations with Generative Adversarial Networks
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Automatic Risk Adaptation in Distributional Reinforcement Learning
• Lineage Evolution Reinforcement Learning
• Quantile Filtered Imitation Learning
• Batch Reinforcement Learning from Crowds
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Distributional Reinforcement Learning for mmWave Communications with Intelligent Reflectors on a UAV
• Amortized Variational Deep Q Network
• Leveraging the Variance of Return Sequences for Exploration Policy
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training
• Efficient Reinforcement Learning Development with RLzoo
• Bayesian Distributional Policy Gradients
• Conditional Importance Sampling for Off-Policy Learning
• Predicting Periodicity with Temporal Difference Learning