Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
arXiv:1910.01249
Abstract
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the predictions of our bounds to the empirical variance in simulation experiments.
arXiv commentAccepted at NeurIPS 2019 Workshop on Optimization Foundations for Reinforcement Learning. 7 pages + 6 pages appendix
References in corpus (6)
- Trust Region Policy Optimization
- Benchmarking Deep Reinforcement Learning for Continuous Control
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
- A Tour of Reinforcement Learning: The View from Continuous Control
- The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint