Rethinking Scanning Strategies with Vision Mamba in Semantic Segmentation of Remote Sensing Imagery: An Experimental Study

Abstract

Deep learning methods, especially Convolutional Neural Networks (CNN) and Vision Transformer (ViT), are frequently employed to perform semantic segmentation of high-resolution remotely sensed images. However, CNNs are constrained by their restricted receptive fields, while ViTs face challenges due to their quadratic complexity. Recently, the Mamba model, featuring linear complexity and a global receptive field, has gained extensive attention for vision tasks. In such tasks, images need to be serialized to form sequences compatible with the Mamba model. Numerous research efforts have explored scanning strategies to serialize images, aiming to enhance the Mamba model's understanding of images. However, the effectiveness of these scanning strategies remains uncertain. In this research, we conduct a comprehensive experimental investigation on the impact of mainstream scanning directions and their combinations on semantic segmentation of remotely sensed images. Through extensive experiments on the LoveDA, ISPRS Potsdam, and ISPRS Vaihingen datasets, we demonstrate that no single scanning strategy outperforms others, regardless of their complexity or the number of scanning directions involved. A simple, single scanning direction is deemed sufficient for semantic segmentation of high-resolution remotely sensed images. Relevant directions for future research are also recommended.

References in corpus (29)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• SpectralGPT: Spectral Remote Sensing Foundation Model
• Cascaded Recurrent Neural Networks for Hyperspectral Image Classification
• Efficiently Modeling Long Sequences with Structured State Spaces
• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
• VMamba: Visual State Space Model
• Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images
• A2-FPN for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
• U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
• VM-UNet: Vision Mamba UNet for Medical Image Segmentation
• LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Advancements in Point Cloud Data Augmentation for Deep Learning: A Survey
• Resurrecting Recurrent Neural Networks for Long Sequences
• SBSS: Stacking-Based Semantic Segmentation Framework for Very High Resolution Remote Sensing Image
• Vivim: a Video Vision Mamba for Medical Video Segmentation
• PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition
• State Space Model for New-Generation Network Alternative to Transformers: A Survey
• RWKV: Reinventing RNNs for the Transformer Era
• Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentation
• SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation
• nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Model
• RS-Mamba for Large Remote Sensing Image Dense Prediction
• RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation
• Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges
• A Survey on Visual Mamba
• Samba: Semantic Segmentation of Remotely Sensed Images with State Space Model

Cited by in corpus (4)

• MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentation
• Seg-LSTM: Performance of xLSTM for Semantic Segmentation of Remotely Sensed Images
• MSCrackMamba: Leveraging Vision Mamba for Crack Detection in Fused Multispectral Imagery
• ClassWise-CRF: Category-Specific Fusion for Enhanced Semantic Segmentation of Remote Sensing Imagery