Reinforcement Learning in POMDP's via Direct Gradient Ascent

Abstract

This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter $\beta \in [0,1)$, which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.

Cited by in corpus (12)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Finding Approximate POMDP solutions Through Belief Compression
• Experiments with Infinite-Horizon, Policy-Gradient Estimation
• A Survey on Applications of Model-Free Strategy Learning in Cognitive Wireless Networks
• Learning from Scarce Experience
• Online Stochastic Optimization under Correlated Bandit Feedback
• Existence of Multiagent Equilibria with Limited Agents
• Hebbian Synaptic Modifications in Spiking Neurons that Learn
• Policy Gradient in Partially Observable Environments: Approximation and Convergence
• Reinforcement Learning applied to Single Neuron
• The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs
• Policy Search with High-Dimensional Context Variables