Online Distillation with Continual Learning for Cyclic Domain Shifts

Abstract

In recent years, online distillation has emerged as a powerful technique for adapting real-time deep neural networks on the fly using a slow, but accurate teacher model. However, a major challenge in online distillation is catastrophic forgetting when the domain shifts, which occurs when the student model is updated with data from the new domain and forgets previously learned knowledge. In this paper, we propose a solution to this issue by leveraging the power of continual learning methods to reduce the impact of domain shifts. Specifically, we integrate several state-of-the-art continual learning methods in the context of online distillation and demonstrate their effectiveness in reducing catastrophic forgetting. Furthermore, we provide a detailed analysis of our proposed solution in the case of cyclic domain shifts. Our experimental results demonstrate the efficacy of our approach in improving the robustness and accuracy of online distillation, with potential applications in domains such as video surveillance or autonomous driving. Overall, our work represents an important step forward in the field of online distillation and continual learning, with the potential to significantly impact real-world applications.

arXiv commentAccepted at the 4th Workshop on Continual Learning in Computer Vision (CVPR 2023)

References in corpus (11)

• Deep Learning in Neural Networks: An Overview
• Overcoming catastrophic forgetting in neural networks
• Prioritized Experience Replay
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Gradient Episodic Memory for Continual Learning
• Tent: Fully Test-time Adaptation by Entropy Minimization
• Online Continual Learning with Maximally Interfered Retrieval
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning
• TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation
• Mixture Domain Adaptation to Improve Semantic Segmentation in Real-World Surveillance
• SimCS: Simulation for Domain Incremental Online Continual Segmentation

Cited by in corpus (3)

• DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics
• Multi-Stream Cellular Test-Time Adaptation of Real-Time Models Evolving in Dynamic Environments
• Physically Interpretable Probabilistic Domain Characterization