Hard Attention Control By Mutual Information Maximization

Abstract

Biological agents have adopted the principle of attention to limit the rate of incoming information from the environment. One question that arises is if an artificial agent has access to only a limited view of its surroundings, how can it control its attention to effectively solve tasks? We propose an approach for learning how to control a hard attention window by maximizing the mutual information between the environment state and the attention location at each step. The agent employs an internal world model to make predictions about its state and focuses attention towards where the predictions may be wrong. Attention is trained jointly with a dynamic memory architecture that stores partial observations and keeps track of the unobserved state. We demonstrate that our approach is effective in predicting the full state from a sequence of partial observations. We also show that the agent's internal representation of the surroundings, a live mental map, can be used for control in two partially observable reinforcement learning tasks. Videos of the trained agent can be found at https://sites.google.com/view/hard-attention-control.

References in corpus (11)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Recurrent Models of Visual Attention
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Episodic Curiosity through Reachability
• Variational Intrinsic Control
• Neural Map: Structured Memory for Deep Reinforcement Learning
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• Memory Augmented Control Networks