CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

Abstract

Self-supervised learning (SSL) has gained widespread attention in the remote sensing (RS) and earth observation (EO) communities owing to its ability to learn task-agnostic representations without human-annotated labels. Nevertheless, most existing RS SSL methods are limited to learning either global semantic separable or local spatial perceptible representations. We argue that this learning strategy is suboptimal in the realm of RS, since the required representations for different RS downstream tasks are often varied and complex. In this study, we proposed a unified SSL framework that is better suited for RS images representation learning. The proposed SSL framework, Contrastive Mask Image Distillation (CMID), is capable of learning representations with both global semantic separability and local spatial perceptibility by combining contrastive learning (CL) with masked image modeling (MIM) in a self-distillation way. Furthermore, our CMID learning framework is architecture-agnostic, which is compatible with both convolutional neural networks (CNN) and vision transformers (ViT), allowing CMID to be easily adapted to a variety of deep learning (DL) applications for RS understanding. Comprehensive experiments have been carried out on four downstream tasks (i.e. scene classification, semantic segmentation, object-detection, and change detection) and the results show that models pre-trained using CMID achieve better performance than other state-of-the-art SSL methods on multiple downstream tasks. The code and pre-trained models will be made available at https://github.com/NJU-LHRS/official-CMID to facilitate SSL research and speed up the development of RS images DL applications.

arXiv commentAccepted by IEEE TGRS. The codes and models are released at https://github.com/NJU-LHRS/official-CMID

References in corpus (31)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• mixup: Beyond Empirical Risk Minimization
• Deep learning in remote sensing: a review
• Remote Sensing Image Scene Classification: Benchmark and State of the Art
• Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• A Survey on Object Detection in Optical Remote Sensing Images
• Remote Sensing Image Change Detection with Transformers
• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community
• MMRotate: A Rotated Object Detection Benchmark using PyTorch
• An Empirical Study of Remote Sensing Pretraining
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images
• Contrastive Masked Autoencoders are Stronger Vision Learners
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
• What Should Not Be Contrastive in Contrastive Learning
• Benchmarking Detection Transfer Learning with Vision Transformers
• Partial Convolution based Padding
• Self-EMD: Self-Supervised Object Detection without ImageNet
• Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
• Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation
• Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling
• Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Masked Siamese ConvNets
• RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?

Cited by in corpus (7)

• A Survey of Sample-Efficient Deep Learning for Change Detection in Remote Sensing: Tasks, Strategies, and Challenges
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
• SSL4SAR: Self-Supervised Learning for Glacier Calving Front Extraction from SAR Imagery
• Ice-FMBench: A Foundation Model Benchmark for Sea Ice Type Segmentation
• Weakly supervised framework for wildlife detection and counting in challenging Arctic environments: a case study on caribou (Rangifer tarandus)
• Self-Supervised Learning for Real-World Object Detection: a Survey