UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery

Abstract

Semantic segmentation of remotely sensed urban scene images is required in a wide range of practical applications, such as land cover mapping, urban change detection, environmental protection, and economic assessment.Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated semantic segmentation for many years. CNN adopts hierarchical feature representation, demonstrating strong capabilities for local information extraction. However, the local property of the convolution layer limits the network from capturing the global context. Recently, as a hot topic in the domain of computer vision, Transformer has demonstrated its great potential in global information modelling, boosting many vision-related tasks such as image classification, object detection, and particularly semantic segmentation. In this paper, we propose a Transformer-based decoder and construct a UNet-like Transformer (UNetFormer) for real-time urban scene segmentation. For efficient segmentation, the UNetFormer selects the lightweight ResNet18 as the encoder and develops an efficient global-local attention mechanism to model both global and local information in the decoder. Extensive experiments reveal that our method not only runs faster but also produces higher accuracy compared with state-of-the-art lightweight models. Specifically, the proposed UNetFormer achieved 67.8% and 52.4% mIoU on the UAVid and LoveDA datasets, respectively, while the inference speed can achieve up to 322.4 FPS with a 512x512 input on a single NVIDIA GTX 3090 GPU. In further exploration, the proposed Transformer-based decoder combined with a Swin Transformer encoder also achieves the state-of-the-art result (91.3% F1 and 84.1% mIoU) on the Vaihingen dataset. The source code will be freely available at https://github.com/WangLibo1995/GeoSeg.

arXiv commentAccepted by ISPRS

References in corpus (17)

• Deep learning in remote sensing: a review
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• Domain-adversarial neural networks to address the appearance variability of histopathology images
• Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models
• Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery using Deep Learning
• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Improving Semantic Segmentation of Aerial Images Using Patch-based Attention
• SCAttNet: Semantic Segmentation Network with Spatial and Channel Attention Mechanism for High-Resolution Remote Sensing Images
• Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models
• An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery
• Road Segmentation for Remote Sensing Images using Adversarial Spatial Pyramid Networks
• Dense Dilated Convolutions Merging Network for Land Cover Classification
• Dynamic Multi-Context Segmentation of Remote Sensing Images based on Convolutional Networks
• Effective Sequential Classifier Training for SVM-based Multitemporal Remote Sensing Image Classification

Cited by in corpus (23)

• Building extraction with vision transformer
• UNetMamba: An Efficient UNet-Like Mamba for Semantic Segmentation of High-Resolution Remote Sensing Images
• HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation
• A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation
• LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing
• Imbalance Knowledge-Driven Multi-modal Network for Land-Cover Semantic Segmentation Using Images and LiDAR Point Clouds
• Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models
• Semantic Labeling of High Resolution Images Using EfficientUNets and Transformers
• MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images
• A large-scale climate-aware satellite image dataset for domain adaptive land-cover semantic segmentation
• DDF: A Novel Dual-Domain Image Fusion Strategy for Remote Sensing Image Semantic Segmentation with Unsupervised Domain Adaptation
• LLM Agent Framework for Intelligent Change Analysis in Urban Environment using Remote Sensing Imagery
• A Multi-scale Generalized Shrinkage Threshold Network for Image Blind Deblurring in Remote Sensing
• Graph Information Bottleneck for Remote Sensing Segmentation
• RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
• AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation
• A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing
• IRSAMap:Towards Large-Scale, High-Resolution Land Cover Map Vectorization
• Learning with less: label-efficient land cover classification at very high spatial resolution using self-supervised deep learning
• Remote Sensing Image Dehazing: A Systematic Review of Progress, Challenges, and Prospects
• A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data
• OmniUnet: A Multimodal Network for Unstructured Terrain Segmentation on Planetary Rovers Using RGB, Depth, and Thermal Imagery
• SegAssess: Panoramic quality mapping for robust and transferable unsupervised segmentation assessment