Recurrent Predictive State Policy Networks

Abstract

We introduce Recurrent Predictive State Policy (RPSP) networks, a recurrent architecture that brings insights from predictive state representations to reinforcement learning in partially observable environments. Predictive state policy networks consist of a recursive filter, which keeps track of a belief about the state of the environment, and a reactive policy that directly maps beliefs to actions, to maximize the cumulative reward. The recursive filter leverages predictive state representations (PSRs) (Rosencrantz and Gordon, 2004; Sun et al., 2016) by modeling predictive state-- a prediction of the distribution of future observations conditioned on history and future actions. This representation gives rise to a rich class of statistically consistent algorithms (Hefny et al., 2018) to initialize the recursive filter. Predictive state serves as an equivalent representation of a belief state. Therefore, the policy component of the RPSP-network can be purely reactive, simplifying training while still allowing optimal behaviour. Moreover, we use the PSR interpretation during training as well, by incorporating prediction error in the loss function. The entire network (recursive filter and reactive policy) is still differentiable and can be trained using gradient based methods. We optimize our policy using a combination of policy gradient based on rewards (Williams, 1992) and gradient descent based on prediction error. We show the efficacy of RPSP-networks under partial observability on a set of robotic control tasks from OpenAI Gym. We empirically show that RPSP-networks perform well compared with memory-preserving networks such as GRUs, as well as finite memory models, being the overall best performing method.

References in corpus (8)

• Playing Atari with Deep Reinforcement Learning
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• End to End Learning for Self-Driving Cars
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Backprop KF: Learning Discriminative Deterministic State Estimators
• Supervised Learning for Dynamical System Learning
• Predictive-State Decoders: Encoding the Future into Recurrent Networks
• Reinforcement Learning of POMDPs using Spectral Methods

Cited by in corpus (5)

• Neural Predictive Belief Representations
• Learning Belief Representations for Imitation Learning in POMDPs
• Sequential Topological Representations for Predictive Models of Deformable Objects
• Online Learning and Planning in Partially Observable Domains without Prior Knowledge
• Tensor Decomposition for Multi-agent Predictive State Representation