Implicit Quantile Networks for Distributional Reinforcement Learning

Abstract

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.

arXiv commentICML 2018

References in corpus (13)

• Dueling Network Architectures for Deep Reinforcement Learning
• Noisy Networks for Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Massively Parallel Methods for Deep Reinforcement Learning
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Wasserstein Auto-Encoders
• Distributional Reinforcement Learning with Quantile Regression
• From optimal transport to generative modeling: the VEGAN cookbook
• Kalman Temporal Differences
• Algorithms for CVaR Optimization in MDPs
• An Analysis of Categorical Distributional Reinforcement Learning
• On the Sample Complexity of Reinforcement Learning with a Generative Model
• Efficient exploration with Double Uncertain Value Networks

Cited by in corpus (75)

• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Challenges of Real-World Reinforcement Learning
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• A Survey of Deep Reinforcement Learning in Video Games
• A Theoretical Analysis of Deep Q-Learning
• ChainerRL: A Deep Reinforcement Learning Library
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Autoregressive Quantile Networks for Generative Modeling
• Mastering Atari with Discrete World Models
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
• Harnessing Structures for Value-Based Planning and Reinforcement Learning
• A case for new neural network smoothness constraints
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• Simplifying Deep Reinforcement Learning via Self-Supervision
• An Open-Source Framework for Adaptive Traffic Signal Control
• Off-Policy Risk Assessment in Contextual Bandits
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning
• Reinforcement Learning-based Switching Controller for a Milliscale Robot in a Constrained Environment
• Adversarial Attacks on Reinforcement Learning based Energy Management Systems of Extended Range Electric Delivery Vehicles
• Affordance-based Reinforcement Learning for Urban Driving
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning with Decorrelation
• Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
• Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Implicit Distributional Reinforcement Learning
• Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results
• Learning to Score Behaviors for Guided Policy Optimization
• Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering
• Risk-Aware Continuous Control with Neural Contextual Bandits
• A Risk-Sensitive Approach to Policy Optimization
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Universal Off-Policy Evaluation
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• Policy Search by Target Distribution Learning for Continuous Control
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
• Learning to Represent Action Values as a Hypergraph on the Action Vertices
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
• Learning Policies through Quantile Regression
• Safe Distributional Reinforcement Learning
• Stochastically Dominant Distributional Reinforcement Learning
• Adapting to Reward Progressivity via Spectral Reinforcement Learning
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods
• A Scheme for Dynamic Risk-Sensitive Sequential Decision Making
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Automatic Risk Adaptation in Distributional Reinforcement Learning
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Large Batch Experience Replay
• Lineage Evolution Reinforcement Learning
• IGN : Implicit Generative Networks
• Two steps to risk sensitivity
• Biased Estimates of Advantages over Path Ensembles
• A Study of Policy Gradient on a Class of Exactly Solvable Models
• Catastrophe, Compounding & Consistency in Choice
• Ranking Policy Gradient
• Active Offline Policy Selection
• From Persistent Homology to Reinforcement Learning with Applications for Retail Banking