Evaluating Online Continual Learning with CALM

Abstract

Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn "on-the-wild". Yet, commonly available benchmarks are far from these real-world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Learning multiple visual domains with residual adapters
• Training Tips for the Transformer Model
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Online Learning: A Comprehensive Survey
• Learning Factored Representations in a Deep Mixture of Experts
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• Online Continual Learning on Sequences
• Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

Cited by in corpus (5)

• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Continual Learning in Recurrent Neural Networks
• Encoders and Ensembles for Task-Free Continual Learning