Clipped Action Policy Gradient

Abstract

Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of actions being clipped to reduce the variance in estimation. We prove that our estimator, named clipped action policy gradient (CAPG), is unbiased and achieves lower variance than the conventional estimator that ignores action bounds. Experimental results demonstrate that CAPG generally outperforms the conventional estimator, indicating that it is a better policy gradient estimator for continuous control tasks. The source code is available at https://github.com/pfnet-research/capg.

arXiv commentAccepted at ICML 2018

References in corpus (2)

• Benchmarking Deep Reinforcement Learning for Continuous Control
• Clipped Action Policy Gradient

Cited by in corpus (13)

• Understanding the impact of entropy on policy optimization
• Trajectory Planning with Deep Reinforcement Learning in High-Level Action Spaces
• Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
• Clipped Action Policy Gradient
• Generalization in Transfer Learning
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• SuperSuit: Simple Microwrappers for Reinforcement Learning Environments
• Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications
• A Risk-Sensitive Approach to Policy Optimization
• Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
• Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• A Contraction Approach to Model-based Reinforcement Learning