Building extraction with vision transformer

Abstract

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.

Submitted to TGRS

References in corpus (15)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Deep learning in remote sensing: a review
• Fully Convolutional Networks for Semantic Segmentation
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery
• Remote Sensing Image Change Detection with Transformers
• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
• A2-FPN for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
• Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Land Cover Classification from Remote Sensing Images Based on Multi-Scale Fully Convolutional Network

Cited by in corpus (4)

• Building Extraction from Remote Sensing Images via an Uncertainty-Aware Network
• Globally Scalable Glacier Mapping by Deep Learning Matches Expert Delineation Accuracy
• Building-road Collaborative Extraction from Remotely Sensed Images via Cross-Interaction
• LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images