A Distributional Perspective on Reinforcement Learning

Abstract

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

ICML 2017

References in corpus (4)

• Dueling Network Architectures for Deep Reinforcement Learning
• Massively Parallel Methods for Deep Reinforcement Learning
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Kalman Temporal Differences

Cited by in corpus (49)

• An Introduction to Deep Reinforcement Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Exploration by Random Network Distillation
• Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Hyperbolic Discounting and Learning over Multiple Horizons
• One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Measuring the Reliability of Reinforcement Learning Algorithms
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
• Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• On Wasserstein Reinforcement Learning and the Fokker-Planck equation
• Temporal Difference Variational Auto-Encoder
• Improving Regression Performance with Distributional Losses
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Reinforcement Learning-based Switching Controller for a Milliscale Robot in a Constrained Environment
• A review of motion planning algorithms for intelligent robotics
• Provable Fictitious Play for General Mean-Field Games
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
• Distributional Advantage Actor-Critic
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning with Decorrelation
• The Potential of the Return Distribution for Exploration in RL
• Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
• Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark
• Trust Region Value Optimization using Kalman Filtering
• Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning
• A Risk-Sensitive Approach to Policy Optimization
• Universal Off-Policy Evaluation
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• On Catastrophic Interference in Atari 2600 Games
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Distributional Reinforcement Learning with Ensembles
• The Barbados 2018 List of Open Issues in Continual Learning
• Reinforcement Learning with Latent Flow
• Regression Analyses of Distributions using Quantile Functional Regression
• Fractal AI: A fragile theory of intelligence
• Learning Risk Preferences in Markov Decision Processes: an Application to the Fourth Down Decision in the National Football League
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation
• A survey of benchmarking frameworks for reinforcement learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Improving On-policy Learning with Statistical Reward Accumulation
• A Theory of Non-Acyclic Generative Flow Networks
• Improved Reinforcement Learning Coordinated Control of a Mobile Manipulator using Joint Clamping
• Solving Atari Games Using Fractals And Entropy