Neural Map: Structured Memory for Deep Reinforcement Learning

Abstract

A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.

References in corpus (3)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Control of Memory, Active Perception, and Action in Minecraft
• Playing Doom with SLAM-Augmented Deep Reinforcement Learning

Cited by in corpus (53)

• On Evaluation of Embodied Navigation Agents
• Object Goal Navigation using Goal-Oriented Semantic Exploration
• A Survey of Deep Reinforcement Learning in Video Games
• Neural SLAM: Learning to Explore with External Memory
• ViZDoom Competitions: Playing Doom from Pixels
• TextWorld: A Learning Environment for Text-based Games
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• Unifying Map and Landmark Based Representations for Visual Navigation
• Learning to Navigate in Cities Without a Map
• Value Prediction Network
• Memory Augmented Control Networks
• VTNet: Visual Transformer Network for Object Goal Navigation
• Active Neural Localization
• A Critical Investigation of Deep Reinforcement Learning for Navigation
• Graph Element Networks: adaptive, structured computation and memory
• Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning
• A Behavioral Approach to Visual Navigation with Graph Localization Networks
• Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents
• Situational Fusion of Visual Representation for Visual Navigation
• Learning models for visual 3D localization with implicit mapping
• To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments
• A Short Survey On Memory Based Reinforcement Learning
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Deep Context Maps: Agent Trajectory Prediction using Location-specific Latent Maps
• Learning and Planning with a Semantic Model
• Simultaneous Mapping and Target Driven Navigation
• Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
• Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals
• Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks
• AlignNet: Unsupervised Entity Alignment
• Graph Attention Memory for Visual Navigation
• Reinforcement Learning-Based Coverage Path Planning with Implicit Cellular Decomposition
• EMPNet: Neural Localisation and Mapping Using Embedded Memory Points
• Where to Look Next: Unsupervised Active Visual Exploration on 360° Input
• Learning Object-conditioned Exploration using Distributed Soft Actor Critic
• Learning to Map for Active Semantic Goal Navigation
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Spatial Memory for Context Reasoning in Object Detection
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• Learning to generate classifiers
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• A short note on the decision tree based neural turing machine
• End-to-End Egospheric Spatial Memory
• Pedestrian Trajectory Prediction with Structured Memory Hierarchies
• A survey of benchmarking frameworks for reinforcement learning
• Memory Augmented Neural Network Adaptive Controller for Strict Feedback Nonlinear Systems
• Egocentric Spatial Memory
• Hard Attention Control By Mutual Information Maximization
• Towards real-world navigation with deep differentiable planners
• Learning Synthetic to Real Transfer for Localization and Navigational Tasks
• Localising In Complex Scenes Using Balanced Adversarial Adaptation