Prototype-based Incremental Few-Shot Semantic Segmentation

Abstract

Semantic segmentation models have two fundamental weaknesses: i) they require large training sets with costly pixel-level annotations, and ii) they have a static output space, constrained to the classes of the training set. Toward addressing both problems, we introduce a new task, Incremental Few-Shot Segmentation (iFSS). The goal of iFSS is to extend a pretrained segmentation model with new classes from few annotated images and without access to old training data. To overcome the limitations of existing models iniFSS, we propose Prototype-based Incremental Few-Shot Segmentation (PIFS) that couples prototype learning and knowledge distillation. PIFS exploits prototypes to initialize the classifiers of new classes, fine-tuning the network to refine its features representation. We design a prototype-based distillation loss on the scores of both old and new class prototypes to avoid overfitting and forgetting, and batch-renormalization to cope with non-i.i.d.few-shot data. We create an extensive benchmark for iFSS showing that PIFS outperforms several few-shot and incremental learning methods in all scenarios.

arXiv commentAccepted at BMVC 2021 (Poster)

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• On First-Order Meta-Learning Algorithms
• OCNet: Object Context Network for Scene Parsing
• Few-Shot Segmentation Propagation with Guided Networks
• Adaptive Masked Proxies for Few-Shot Segmentation