Unsupervised Predictive Memory in a Goal-Directed Agent

Abstract

Animals execute goal-directed behaviours despite the limited range and scope of their sensors. To cope, they explore environments and store memories maintaining estimates of important information that is not presently available. Recently, progress has been made with artificial intelligence (AI) agents that learn to perform tasks from sensory input, even at a human level, by merging reinforcement learning (RL) algorithms with deep neural networks, and the excitement surrounding these results has led to the pursuit of related ideas as explanations of non-human animal learning. However, we demonstrate that contemporary RL algorithms struggle to solve simple tasks when enough information is concealed from the sensors of the agent, a property called "partial observability". An obvious requirement for handling partially observed tasks is access to extensive memory, but we show memory is not enough; it is critical that the right information be stored in the right format. We develop a model, the Memory, RL, and Inference Network (MERLIN), in which memory formation is guided by a process of predictive modeling. MERLIN facilitates the solution of tasks in 3D virtual reality environments for which partial observability is severe and memories must be maintained over long durations. Our model demonstrates a single learning agent architecture that can solve canonical behavioural tasks in psychology and neurobiology without strong simplifying assumptions about the dimensionality of sensory input or the duration of experiences.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A guide to convolution arithmetic for deep learning
• Learning to reinforcement learn
• One-Shot Imitation Learning
• Control of Memory, Active Perception, and Action in Minecraft
• Grounded Language Learning in a Simulated 3D World
• Generative Temporal Models with Memory

Cited by in corpus (54)

• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Stabilizing Transformers for Reinforcement Learning
• Hierarchical visuomotor control of humanoids
• Decoupling Representation Learning from Reinforcement Learning
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Meta reinforcement learning as task inference
• Learning to Navigate in Cities Without a Map
• Neural Predictive Belief Representations
• Grounded Language Learning Fast and Slow
• Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversal
• Improving the Gating Mechanism of Recurrent Neural Networks
• Mastering Atari with Discrete World Models
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Offline Reinforcement Learning from Images with Latent Space Models
• Learning models for visual 3D localization with implicit mapping
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• A Short Survey On Memory Based Reinforcement Learning
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Synthetic Returns for Long-Term Credit Assignment
• Meta-Learning surrogate models for sequential decision making
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
• If deep learning is the answer, then what is the question?
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• AHA! an 'Artificial Hippocampal Algorithm' for Episodic Machine Learning
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers
• Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
• Product Kanerva Machines: Factorized Bayesian Memory
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Unsupervised Emergence of Egocentric Spatial Structure from Sensorimotor Prediction
• Cross-View Policy Learning for Street Navigation
• Model-Based Episodic Memory Induces Dynamic Hybrid Controls
• The Principle of Unchanged Optimality in Reinforcement Learning Generalization
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Improving Reliable Navigation under Uncertainty via Predictions Informed by Non-Local Information
• State representation learning with recurrent capsule networks
• Unsupervised Emergence of Spatial Structure from Sensorimotor Prediction
• The neural and cognitive architecture for learning from a small sample
• Intrinsic motivations and open-ended learning
• Data-Efficient Policy Selection for Navigation in Partial Maps via Subgoal-Based Abstraction
• End-to-End Egospheric Spatial Memory
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer Architecture
• Amanuensis: The Programmer's Apprentice
• Planning with Brain-inspired AI
• Rapid Task-Solving in Novel Environments
• Graph Convolutional Memory using Topological Priors
• Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning
• Hybrid Active Inference
• Multitask Adaptation by Retrospective Exploration with Learned World Models
• A Proposal for Intelligent Agents with Episodic Memory