Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data

Abstract

In spite of remarkable success of the convolutional neural networks on semantic segmentation, they suffer from catastrophic forgetting: a significant performance drop for the already learned classes when new classes are added on the data, having no annotations for the old classes. We propose an incremental learning methodology, enabling to learn segmenting new classes without hindering dense labeling abilities for the previous classes, although the entire previous data are not accessible. The key points of the proposed approach are adapting the network to learn new as well as old classes on the new training data, and allowing it to remember the previously learned information for the old classes. For adaptation, we keep a frozen copy of the previously trained network, which is used as a memory for the updated network in absence of annotations for the former classes. The updated network minimizes a loss function, which balances the discrepancy between outputs for the previous classes from the memory and updated networks, and the mis-classification rate between outputs for the new classes from the updated network and the new ground-truth. For remembering, we either regularly feed samples from the stored, little fraction of the previous data or use the memory network, depending on whether the new data are collected from completely different geographic areas or from the same city. Our experimental results prove that it is possible to add new classes to the network, while maintaining its performance for the previous classes, despite the whole previous training data are not available.

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Progressive Neural Networks
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Gradient Episodic Memory for Continual Learning
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks
• Variational Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Encoder Based Lifelong Learning
• Reinforced Continual Learning
• High-Resolution Semantic Labeling with Convolutional Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing
• Incremental Classifier Learning with Generative Adversarial Networks
• Task Agnostic Continual Learning Using Online Variational Bayes
• SupportNet: solving catastrophic forgetting in class incremental learning with support data
• Active Long Term Memory Networks

Cited by in corpus (22)

• Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation
• DAugNet: Unsupervised, Multi-source, Multi-target, and Life-long Domain Adaptation for Semantic Segmentation of Satellite Images
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Recent Advances of Continual Learning in Computer Vision: An Overview
• ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• RSI-Net: Two-Stream Deep Neural Network for Remote Sensing Imagesbased Semantic Segmentation
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Incremental Learning for Multi-organ Segmentation with Partially Labeled Datasets
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• Modeling the Background for Incremental and Weakly-Supervised Semantic Segmentation
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• ContCap: A scalable framework for continual image captioning
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation
• Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
• On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances and Million-AID
• Interactive Learning for Semantic Segmentation in Earth Observation
• Targeted Gradient Descent: A Novel Method for Convolutional Neural Networks Fine-tuning and Online-learning
• Reviewing continual learning from the perspective of human-level intelligence
• Towards Recognizing New Semantic Concepts in New Visual Domains
• Explaining How Deep Neural Networks Forget by Deep Visualization
• An EM Framework for Online Incremental Learning of Semantic Segmentation