BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval

Abstract

This paper presents the multi-modal BigEarthNet (BigEarthNet-MM) benchmark archive made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches to support the deep learning (DL) studies in multi-modal multi-label remote sensing (RS) image retrieval and classification. Each pair of patches in BigEarthNet-MM is annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its thematically most detailed Level-3 class nomenclature. Our initial research demonstrates that some CLC classes are challenging to be accurately described by only considering (single-date) BigEarthNet-MM images. In this paper, we also introduce an alternative class-nomenclature as an evolution of the original CLC labels to address this problem. This is achieved by interpreting and arranging the CLC Level-3 nomenclature based on the properties of BigEarthNet-MM images in a new nomenclature of 19 classes. In our experiments, we show the potential of BigEarthNet-MM for multi-modal multi-label image retrieval and classification problems by considering several state-of-the-art DL models. We also demonstrate that the DL models trained from scratch on BigEarthNet-MM outperform those pre-trained on ImageNet, especially in relation to some complex classes, including agriculture and other vegetated and natural environments. We make all the data and the DL models publicly available at https://bigearth.net, offering an important resource to support studies on multi-modal image scene classification and retrieval problems in RS.

arXiv commentAccepted at the IEEE Geoscience and Remote Sensing Magazine. Our code is available online at https://git.tu-berlin.de/rsim/BigEarthNet-MM_19-classes_models. arXiv admin note: substantial text overlap with arXiv:2001.06372

References in corpus (4)

• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities
• BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding
• Metric-Learning based Deep Hashing Network for Content Based Retrieval of Remote Sensing Images

Cited by in corpus (25)

• Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture
• On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification
• There Are No Data Like More Data- Datasets for Deep Learning in Earth Observation
• Multi-Label Noise Robust Collaborative Learning for Remote Sensing Image Classification
• Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data
• SpectralEarth: Training Hyperspectral Foundation Models at Scale
• Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications
• Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classification
• A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing
• A generic self-supervised learning (SSL) framework for representation learning from spectra-spatial feature of unlabeled remote sensing imagery
• Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
• Generative Reasoning Integrated Label Noise Robust Deep Image Representation Learning
• A Consensual Collaborative Learning Method for Remote Sensing Image Classification Under Noisy Multi-Labels
• Spotting Virus from Satellites: Modeling the Circulation of West Nile Virus Through Graph Neural Networks
• Fus-MAE: A cross-attention-based data fusion approach for Masked Autoencoders in remote sensing
• Joint Source-and-Channel Coding for Small Satellite Applications
• MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning
• A Novel Framework to Jointly Compress and Index Remote Sensing Images for Efficient Content-Based Retrieval
• ViGEO: an Assessment of Vision GNNs in Earth Observation
• Adaptive Gradient Calibration for Single-Positive Multi-Label Learning in Remote Sensing Image Scene Classification
• Informative and Representative Triplet Selection for Multilabel Remote Sensing Image Retrieval
• Survey on Remote Sensing Scene Classification: From Traditional Methods to Large Generative AI Models