Experience Replay for Continual Learning

Abstract

Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

NeurIPS 2019

References in corpus (6)

• Overcoming catastrophic forgetting in neural networks
• Continual Learning Through Synaptic Intelligence
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Q-learning from Demonstrations
• Progress & Compress: A scalable framework for continual learning
• Policy Consolidation for Continual Reinforcement Learning

Cited by in corpus (47)

• A continual learning survey: Defying forgetting in classification tasks
• On Tiny Episodic Memories in Continual Learning
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Federated Continual Learning via Knowledge Fusion: A Survey
• Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting
• Functional Regularisation for Continual Learning with Gaussian Processes
• Online Continual Learning with Maximally Interfered Retrieval
• AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning
• Continual Learning and Catastrophic Forgetting
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• Orthogonal Gradient Descent for Continual Learning
• Continual Learning in Low-rank Orthogonal Subspaces
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• Continual Learning with Adaptive Weights (CLAW)
• A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment
• Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition
• Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
• AirLoop: Lifelong Loop Closure Detection
• Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning
• Linear Mode Connectivity in Multitask and Continual Learning
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
• Continual Learning for Monolingual End-to-End Automatic Speech Recognition
• Adaptive Online Planning for Continual Lifelong Learning
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization
• Powerpropagation: A sparsity inducing weight reparameterisation
• MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning
• Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
• A Study of Continual Learning Methods for Q-Learning
• Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
• Wide Neural Networks Forget Less Catastrophically
• Confidence-Guided Learning Process for Continuous Classification of Time Series
• Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
• Review learning: Real world validation of privacy preserving continual learning across medical institutions
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Continual Learning in the Presence of Repetition
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• Towards Combining On-Off-Policy Methods for Real-World Applications
• Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines
• Domain-Aware Augmentations for Unsupervised Online General Continual Learning
• Learning Barrier Functions with Memory for Robust Safe Navigation
• Continual Novelty Detection
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Center Loss Regularization for Continual Learning
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay
• Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning