Weighted Ensemble Models Are Strong Continual Learners

Abstract

In this work, we study the problem of continual learning (CL) where the goal is to learn a model on a sequence of tasks, such that the data from the previous tasks becomes unavailable while learning on the current task data. CL is essentially a balancing act between being able to learn on the new task (i.e., plasticity) and maintaining the performance on the previously learned concepts (i.e., stability). Intending to address the stability-plasticity trade-off, we propose to perform weight-ensembling of the model parameters of the previous and current tasks. This weighted-ensembled model, which we call Continual Model Averaging (or CoMA), attains high accuracy on the current task by leveraging plasticity, while not deviating too far from the previous weight configuration, ensuring stability. We also propose an improved variant of CoMA, named Continual Fisher-weighted Model Averaging (or CoFiMA), that selectively weighs each parameter in the weights ensemble by leveraging the Fisher information of the weights of the model. Both variants are conceptually simple, easy to implement, and effective in attaining state-of-the-art performance on several standard CL benchmarks. Code is available at: https://github.com/IemProg/CoFiMA.

Accepted for ECCV2024, Code: https://github.com/IemProg/CoFiMA

References in corpus (27)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Progressive Neural Networks
• DINOv2: Learning Robust Visual Features without Supervision
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• Three scenarios for continual learning
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• What is being transferred in transfer learning?
• Revisiting Natural Gradient for Deep Networks
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence
• Functional Regularisation for Continual Learning with Gaussian Processes
• S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• An Empirical Investigation of the Role of Pre-training in Lifelong Learning
• Patching open-vocabulary models by interpolating weights
• RanPAC: Random Projections and Pre-trained Models for Continual Learning
• Continual Normalization: Rethinking Batch Normalization for Online Continual Learning
• Fusing finetuned models for better pretraining
• Divide and not forget: Ensemble of selectively trained experts in Continual Learning
• ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning
• A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model
• Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need