Learning Policies with External Memory

Abstract

In order for an agent to perform well in partially observable domains, it is usually necessary for actions to depend on the history of observations. In this paper, we explore a {\it stigmergic} approach, in which the agent's actions include the ability to set and clear bits in an external memory, and the external memory is included as part of the input to the agent. In this case, we need to learn a reactive policy in a highly non-Markovian domain. We explore two algorithms: SARSA(λ), which has had empirical success in partially observable domains, and VAPS, a new algorithm due to Baird and Moore, with convergence guarantees in partially observable domains. We compare the performance of these two algorithms on benchmark problems.

8 pages

References in corpus (3)

• Solving POMDPs by Searching in Policy Space
• Learning Finite-State Controllers for Partially Observable Environments
• Solving POMDPs by Searching the Space of Finite Policies

Cited by in corpus (13)

• Learning to Cooperate via Policy Search
• Memory-based control with recurrent neural networks
• Learning Finite-State Controllers for Partially Observable Environments
• Solving POMDPs by Searching the Space of Finite Policies
• Nonapproximability Results for Partially Observable Markov Decision Processes
• Learning from Scarce Experience
• Policy Improvement for POMDPs Using Normalized Importance Sampling
• Stigmergic Independent Reinforcement Learning for Multi-Agent Collaboration
• Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning
• The Geometry of Memoryless Stochastic Policy Optimization in Infinite-Horizon POMDPs
• Bounds on sample size for policy evaluation in Markov environments
• How memory architecture affects learning in a simple POMDP: the two-hypothesis testing problem
• Learning to Cooperate via Policy Search