Dark Experience for General Continual Learning: a Strong, Simple Baseline

Abstract

Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.

24 pages, 4 figures. Accepted at 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• On Calibration of Modern Neural Networks
• A continual learning survey: Defying forgetting in classification tasks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Efficient Lifelong Learning with A-GEM
• Learning with Pseudo-Ensembles
• On Tiny Episodic Memories in Continual Learning
• Three Factors Influencing Minima in SGD
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Towards Robust Evaluations of Continual Learning
• Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Cited by in corpus (15)

• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• Cost-effective On-device Continual Learning over Memory Hierarchy with Miro
• Online Lifelong Generalized Zero-Shot Learning
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification
• MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Task-agnostic Continual Learning with Hybrid Probabilistic Models
• Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning
• Lifelong Learning for Neural powered Mixed Integer Programming
• Continual Learning in Deep Networks: an Analysis of the Last Layer
• Multi-grained spatial-temporal feature complementarity for accurate online cellular traffic prediction
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• Class Incremental Online Streaming Learning