Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Abstract

Continual learning (CL) breaks off the one-way training manner and enables a model to adapt to new data, semantics and tasks continuously. However, current CL methods mainly focus on single tasks. Besides, CL models are plagued by catastrophic forgetting and semantic drift since the lack of old data, which often occurs in remote-sensing interpretation due to the intricate fine-grained semantics. In this paper, we propose Continual Panoptic Perception (CPP), a unified continual learning model that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception for universal interpretation in remote sensing images. Concretely, we propose a collaborative cross-modal encoder (CCE) to extract the input image features, which supports pixel classification and caption generation synchronously. To inherit the knowledge from the old model without exemplar memory, we propose a task-interactive knowledge distillation (TKD) method, which leverages cross-modal optimization and task-asymmetric pseudo-labeling (TPL) to alleviate catastrophic forgetting. Furthermore, we also propose a joint optimization mechanism to achieve end-to-end multi-modal panoptic perception. Experimental results on the fine-grained panoptic perception dataset validate the effectiveness of the proposed model, and also prove that joint optimization can boost sub-task CL efficiency with over 13\% relative improvement on panoptic quality.

arXiv commentAccepted in ACMMM 2024

References in corpus (10)

• Per-Pixel Classification is Not All You Need for Semantic Segmentation
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning
• Birds of A Feather Flock Together: Category-Divergence Guidance for Domain Adaptive Segmentation
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation
• Continual Barlow Twins: continual self-supervised learning for remote sensing semantic segmentation
• Improving Replay-Based Continual Semantic Segmentation with Smart Data Selection