Influence-aware Memory Architectures for Deep Reinforcement Learning
arXiv:1911.07643
Abstract
Due to its perceptual limitations, an agent may have too little information about the state of the environment to act optimally. In such cases, it is important to keep track of the observation history to uncover hidden state. Recent deep reinforcement learning methods use recurrent neural networks (RNN) to memorize past observations. However, these models are expensive to train and have convergence difficulties, especially when dealing with high dimensional input spaces. In this paper, we propose influence-aware memory (IAM), a theoretically inspired memory architecture that tries to alleviate the training difficulties by restricting the input of the recurrent layers to those variables that influence the hidden state information. Moreover, as opposed to standard RNNs, in which every piece of information used for estimating Q values is inevitably fed back into the network for the next prediction, our model allows information to flow without being necessarily stored in the RNN's internal memory. Results indicate that, by letting the recurrent layers focus on a small fraction of the observation variables while processing the rest of the information with a feedforward neural network, we can outperform standard recurrent architectures both in training speed and policy performance. This approach also reduces runtime and obtains better scores than methods that stack multiple observations to remove partial observability.
References in corpus (8)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
- Effective Approaches to Attention-based Neural Machine Translation
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- PEGASUS: A Policy Search Method for Large MDPs and POMDPs
- Playing FPS Games with Deep Reinforcement Learning
- Deep Attention Recurrent Q-Network
- On Improving Deep Reinforcement Learning for POMDPs