SpectralGPT: Spectral Remote Sensing Foundation Model
arXiv:2311.07113 · doi:10.1109/TPAMI.2024.3362475
Abstract
The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.
arXiv commentAccepted by IEEE TPAMI
References in corpus (4)
Cited by in corpus (32)
- SpectralGPT: Spectral Remote Sensing Foundation Model
- A Comprehensive Survey for Hyperspectral Image Classification: The Evolution from Conventional to Transformers and Mamba Models
- SMamba: A Spatial-spectral State Space Model for Hyperspectral Image Classification
- Rethinking Scanning Strategies with Vision Mamba in Semantic Segmentation of Remote Sensing Imagery: An Experimental Study
- Triple-domain Feature Learning with Frequency-aware Memory Enhancement for Moving Infrared Small Target Detection
- Spatial and Spatial-Spectral Morphological Mamba for Hyperspectral Image Classification
- WaveMamba: Spatial-Spectral Wavelet Mamba for Hyperspectral Image Classification
- SpectralEarth: Training Hyperspectral Foundation Models at Scale
- Pyramid Hierarchical Transformer for Hyperspectral Image Classification
- A Spectral Diffusion Prior for Hyperspectral Image Super-Resolution
- Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
- When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System
- Better, Not Just More: Data-Centric Machine Learning for Earth Observation
- GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation
- EfficientCD: A New Strategy For Change Detection Based With Bi-temporal Layers Exchanged
- Semi-supervised Semantic Segmentation for Remote Sensing Images via Multi-scale Uncertainty Consistency and Cross-Teacher-Student Attention
- DDF: A Novel Dual-Domain Image Fusion Strategy for Remote Sensing Image Semantic Segmentation with Unsupervised Domain Adaptation
- Content-driven Magnitude-Derivative Spectrum Complementary Learning for Hyperspectral Image Classification
- Transformers Fusion across Disjoint Samples for Hyperspectral Image Classification
- Satellite Edge Artificial Intelligence with Large Models: Architectures and Technologies
- BihoT: A Large-Scale Dataset and Benchmark for Hyperspectral Camouflaged Object Tracking
- Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
- RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models
- Towards Natural Machine Unlearning
- On the Adversarial Vulnerabilities of Transfer Learning in Remote Sensing
- Learning with less: label-efficient land cover classification at very high spatial resolution using self-supervised deep learning
- Importance of Disjoint Sampling in Conventional and Transformer Models for Hyperspectral Image Classification
- Multi-encoder ConvNeXt Network with Smooth Attentional Feature Fusion for Multispectral Semantic Segmentation
- Pi-transformer: A prior-informed dual-attention model for multivariate time-series anomaly detection
- CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities
- Ice-FMBench: A Foundation Model Benchmark for Sea Ice Type Segmentation
- SE-Det: A Strict Spatial-Spectral Equivariant Detector for Multispectral Object Detection