Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

Abstract

Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. Replay Memories are a common solution to the problem, decorrelating and shuffling old and new training samples. They naively store state transitions as they come in, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our paper shows that map-based experience replay allows for significant memory reduction with only small performance decreases.

References in corpus (13)

• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• End-to-End Training of Deep Visuomotor Policies
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Tune: A Research Platform for Distributed Model Selection and Training
• Experience Replay for Continual Learning
• Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty
• Temporal Difference Learning for Model Predictive Control
• Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space
• Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2
• Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning

Cited by in corpus (2)

• Towards Incremental Learning in Large Language Models: A Critical Review
• Growable and Interpretable Neural Control with Online Continual Learning for Autonomous Lifelong Locomotion Learning Machines