Bayesian Policy Gradients via Alpha Divergence Dropout Inference

Abstract

Policy gradient methods have had great success in solving continuous control tasks, yet the stochastic nature of such problems makes deterministic value estimation difficult. We propose an approach which instead estimates a distribution by fitting the value function with a Bayesian Neural Network. We optimize an $\alpha$-divergence objective with Bayesian dropout approximation to learn and estimate this distribution. We show that using the Monte Carlo posterior mean of the Bayesian value function distribution, rather than a deterministic network, improves stability and performance of policy gradient methods in continuous control MuJoCo simulations.

Accepted to Bayesian Deep Learning Workshop at NIPS 2017

References in corpus (7)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Weight Uncertainty in Neural Networks
• Deep Exploration via Bootstrapped DQN
• Parameter Space Noise for Exploration
• A Distributional Perspective on Reinforcement Learning
• Bayesian policy gradient and actor-critic algorithms

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Reward Estimation for Variance Reduction in Deep Reinforcement Learning
• Leveraging exploration in off-policy algorithms via normalizing flows
• Exploration by Distributional Reinforcement Learning
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• The Potential of the Return Distribution for Exploration in RL