Policy Optimization with Second-Order Advantage Information
arXiv:1805.03586
Abstract
Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework to reduce the variance. To invoke RB, our proposed algorithm (POSA) learns the underlying factorization structure among the action space based on the second-order advantage information. POSA captures the quadratic information explicitly and efficiently by utilizing the wide & deep architecture. Empirical studies show that our proposed approach demonstrates the performance improvements on high-dimensional synthetic settings and OpenAI Gym's MuJoCo continuous control tasks.
arXiv commentInternational Joint Conference on Artificial Intelligence (IJCAI) 2018
References in corpus (8)
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- End-to-End Training of Deep Visuomotor Policies
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- Gradient Estimators for Implicit Models
- The Mirage of Action-Dependent Baselines in Reinforcement Learning