Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control

Abstract

Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleatoric uncertainty. Disentangling and evaluating these uncertainties simultaneously stands a chance of improving the agent's final performance, accelerating training, and facilitating quality assurance after deployment. In this work, we propose an uncertainty-aware reinforcement learning algorithm for continuous control tasks that extends the Deep Deterministic Policy Gradient algorithm (DDPG). It exploits epistemic uncertainty to accelerate exploration and aleatoric uncertainty to learn a risk-sensitive policy. We conduct numerical experiments showing that our variant of DDPG outperforms vanilla DDPG without uncertainty estimation in benchmark tasks on robotic control and power-grid optimization.

arXiv comment10 pages, 6 figures. Accepted to International Joint Conference on Neural Networks (IJCNN 2022), July 18-23, Padua, Italy

References in corpus (28)

• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• A Practical Guide to Robust Optimization
• Deep Exploration via Bootstrapped DQN
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Distributed Distributional Deterministic Policy Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Randomized Prior Functions for Deep Reinforcement Learning
• UCB Exploration via Q-Ensembles
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Parametric Return Density Estimation for Reinforcement Learning
• Uncertainty in Neural Networks: Approximately Bayesian Ensembling
• Worst Cases Policy Gradients
• DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning
• Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
• Learning to Run with Actor-Critic Ensemble
• Risk-Averse Offline Reinforcement Learning
• PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems
• Improving Robustness via Risk Averse Distributional Reinforcement Learning
• Distributional Advantage Actor-Critic
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity
• Sample-based Distributional Policy Gradient
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods