Model-Free Episodic Control

Abstract

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.

References in corpus (3)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Prioritized Experience Replay
• Approximate Hubel-Wiesel Modules and the Data Structures of Neural Computation

Cited by in corpus (57)

• Episodic Curiosity through Reachability
• A Survey of Deep Reinforcement Learning in Video Games
• Stabilizing Transformers for Reinforcement Learning
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• A review on deep reinforcement learning for fluid mechanics: an update
• A review on Deep Reinforcement Learning for Fluid Mechanics
• Neural Episodic Control
• Never Give Up: Learning Directed Exploration Strategies
• Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study
• Small Sample Learning in Big Data Era
• Semi-parametric Topological Memory for Navigation
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• Generative Adversarial Self-Imitation Learning
• Composable Planning with Attributes
• One-shot learning and behavioral eligibility traces in sequential decision making
• Been There, Done That: Meta-Learning with Episodic Recall
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Generalizable Episodic Memory for Deep Reinforcement Learning
• Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update
• Learning to Learn Variational Semantic Memory
• Training Agents using Upside-Down Reinforcement Learning
• A Short Survey On Memory Based Reinforcement Learning
• Concurrent Meta Reinforcement Learning
• On the Estimation Bias in Double Q-Learning
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Offline Reinforcement Learning with Value-based Episodic Memory
• Spatial Positioning Token (SPToken) for Smart Mobility
• TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning
• Building Machines that Learn and Think for Themselves: Commentary on Lake et al., Behavioral and Brain Sciences, 2017
• Episodic Memory Deep Q-Networks
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Imitation Learning from Pixel-Level Demonstrations by HashReward
• Model-Based Episodic Memory Induces Dynamic Hybrid Controls
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control
• Memory-Efficient Episodic Control Reinforcement Learning with Dynamic Online k-means
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• Model-Free Episodic Control with State Aggregation
• Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments
• Random Projection in Neural Episodic Control
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Learning Diverse Policies with Soft Self-Generated Guidance
• Learning in the Machine: the Symmetries of the Deep Learning Channel
• Memory and attention in deep learning
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Faster Deep Q-learning using Neural Episodic Control
• Long Timescale Credit Assignment in NeuralNetworks with External Memory
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation
• Multitask Adaptation by Retrospective Exploration with Learned World Models
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Rapid Task-Solving in Novel Environments
• Successor Feature Neural Episodic Control
• Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learning