Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting

Abstract

Neural networks can achieve excellent results in a wide variety of applications. However, when they attempt to sequentially learn, they tend to learn the new task while catastrophically forgetting previous ones. We propose a model that overcomes catastrophic forgetting in sequential reinforcement learning by combining ideas from continual learning in both the image classification domain and the reinforcement learning domain. This model features a dual memory system which separates continual learning from reinforcement learning and a pseudo-rehearsal system that "recalls" items representative of previous tasks via a deep generative network. Our model sequentially learns Atari 2600 games without demonstrating catastrophic forgetting and continues to perform above human level on all three games. This result is achieved without: demanding additional storage requirements as the number of tasks increases, storing raw data or revisiting past tasks. In comparison, previous state-of-the-art solutions are substantially more vulnerable to forgetting on these complex deep reinforcement learning tasks.

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Understanding Neural Networks Through Deep Visualization
• Progressive Neural Networks
• Experience Replay for Continual Learning
• FearNet: Brain-Inspired Model for Incremental Learning
• Progress & Compress: A scalable framework for continual learning
• Deep Generative Dual Memory Network for Continual Learning
• Revisiting Natural Gradient for Deep Networks
• Policy Distillation
• Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks
• On the Limitations of First-Order Approximation in GAN Dynamics
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Two geometric input transformation methods for fast online reinforcement learning with neural nets
• Continual Learning Using World Models for Pseudo-Rehearsal

Cited by in corpus (12)

• Deep Reinforcement and InfoMax Learning
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Continuous learning of spiking networks trained with local rules
• Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Locally Weighted Regression Pseudo-Rehearsal for Online Learning of Vehicle Dynamics
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization
• Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning
• Continual Learning Using World Models for Pseudo-Rehearsal
• A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learning
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• GRIm-RePR: Prioritising Generating Important Features for Pseudo-Rehearsal