Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy

Abstract

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by the recent ARM gradient estimator for discrete random variable models. We show that the ARM policy gradient estimator achieves variance reduction with theoretical guarantees, and leads to significantly more stable and faster convergence of policies parameterized by neural networks.

References in corpus (5)

• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks