Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning

Abstract

Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.

arXiv commentIn Workshop on Continual Learning in Computer Vision at CVPR 2021

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• A continual learning survey: Defying forgetting in classification tasks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Contrastive Representation Learning: A Framework and Review
• Efficient Lifelong Learning with A-GEM
• Contrastive Multiview Coding
• Contrastive Multi-View Representation Learning on Graphs
• On Tiny Episodic Memories in Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Learning Video Representations using Contrastive Bidirectional Transformer
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Regularization Shortcomings for Continual Learning