Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

Abstract

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

Accepted to ICLR 2018, Oral (2%)

References in corpus (3)

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Guided Policy Search as Approximate Mirror Descent

Cited by in corpus (33)

• Deep Reinforcement Learning for Cyber Security
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• A Game Theoretic Framework for Model Based Reinforcement Learning
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• TD-Regularized Actor-Critic Methods
• Settling the Variance of Multi-Agent Policy Gradients
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Counterfactual Data Augmentation using Locally Factored Dynamics
• History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms
• Deep Reinforcement Learning for Dynamic Urban Transportation Problems
• Hindsight Trust Region Policy Optimization
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
• Reinforcement Learning for Uplift Modeling
• Comparing Deep Reinforcement Learning Algorithms in Two-Echelon Supply Chains
• Beyond variance reduction: Understanding the true impact of baselines on policy optimization
• From Importance Sampling to Doubly Robust Policy Gradient
• Variance Reduction for Evolution Strategies via Structured Control Variates
• Reward-estimation variance elimination in sequential decision processes
• Learning Value Functions in Deep Policy Gradients using Residual Variance
• Statistically Efficient Off-Policy Policy Gradients
• Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods
• On Proximal Policy Optimization's Heavy-tailed Gradients
• Soft Actor-Critic With Integer Actions
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• Evaluating a Generative Adversarial Framework for Information Retrieval
• Policy Optimization with Second-Order Advantage Information
• Discrete Action On-Policy Learning with Action-Value Critic
• KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning
• Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning