ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data

Abstract

Scene understanding of high resolution aerial images is of great importance for the task of automated monitoring in various remote sensing applications. Due to the large within-class and small between-class variance in pixel values of objects of interest, this remains a challenging task. In recent years, deep convolutional neural networks have started being used in remote sensing applications and demonstrate state of the art performance for pixel level classification of objects. \textcolor{black}{Here we propose a reliable framework for performant results for the task of semantic segmentation of monotemporal very high resolution aerial images. Our framework consists of a novel deep learning architecture, ResUNet-a, and a novel loss function based on the Dice loss. ResUNet-a uses a UNet encoder/decoder backbone, in combination with residual connections, atrous convolutions, pyramid scene parsing pooling and multi-tasking inference. ResUNet-a infers sequentially the boundary of the objects, the distance transform of the segmentation mask, the segmentation mask and a colored reconstruction of the input. Each of the tasks is conditioned on the inference of the previous ones, thus establishing a conditioned relationship between the various tasks, as this is described through the architecture's computation graph. We analyse the performance of several flavours of the Generalized Dice loss for semantic segmentation, and we introduce a novel variant loss function for semantic segmentation of objects that has excellent convergence properties and behaves well even under the presence of highly imbalanced classes.} The performance of our modeling framework is evaluated on the ISPRS 2D Potsdam dataset. Results show state-of-the-art performance with an average F1 score of 92.9\% over all classes for our best model.

Accepted for publication to the ISPRS Journal of Photogrammetry and Remote Sensing

References in corpus (31)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Deep Residual Learning for Image Recognition
• Deep learning in remote sensing: a review
• Road Extraction by Deep Residual U-Net
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• An Overview of Multi-Task Learning in Deep Neural Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Densely Connected Convolutional Networks
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Wide Residual Networks
• Focal Loss for Dense Object Detection
• Fully Convolutional Networks for Semantic Segmentation
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Boundary loss for highly unbalanced segmentation
• Horovod: fast and easy distributed deep learning in TensorFlow
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks
• Holistically-Nested Edge Detection
• Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network
• Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery
• Identity Mappings in Deep Residual Networks
• Context Encoding for Semantic Segmentation
• The Importance of Skip Connections in Biomedical Image Segmentation
• Rethinking ImageNet Pre-training
• Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping
• Demystifying ResNet
• Semantic Segmentation with Boundary Neural Fields

Cited by in corpus (66)

• UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery
• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
• DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
• MTU-Net: Multi-level TransUNet for Space-based Infrared Tiny Ship Detection
• CSWin-UNet: Transformer UNet with Cross-Shaped Windows for Medical Image Segmentation
• Using DUCK-Net for Polyp Image Segmentation
• WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
• Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images
• Novel DeepONet architecture to predict stresses in elastoplastic structures with variable complex geometries and loads
• CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentation
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation
• A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentation
• Consistency-Regularized Region-Growing Network for Semantic Segmentation of Urban Scenes with Point-Level Annotations
• Multi-task fully convolutional network for tree species mapping in dense forests using small training hyperspectral data
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties
• A global method to identify trees outside of closed-canopy forests with medium-resolution satellite imagery
• Assessment of a new GeoAI foundation model for flood inundation mapping
• CFFormer: Cross CNN-Transformer Channel Attention and Spatial Feature Fusion for Improved Segmentation of Heterogeneous Medical Images
• MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images
• Cleaning our own Dust: Simulating and Separating Galactic Dust Foregrounds with Neural Networks
• SKA Science Data Challenge 2: analysis and results
• RRCANet: Recurrent Reusable-Convolution Attention Network for Infrared Small Target Detection
• SU-Net: Pose estimation network for non-cooperative spacecraft on-orbit
• Bridging the Domain Gap: A Simple Domain Matching Method for Reference-based Image Super-Resolution in Remote Sensing
• GLIMS: Attention-Guided Lightweight Multi-Scale Hybrid Network for Volumetric Semantic Segmentation
• UMat: Uncertainty-Aware Single Image High Resolution Material Capture
• Deep Model-Based Super-Resolution with Non-uniform Blur
• MNet-SAt: A Multiscale Network with Spatial-enhanced Attention for Segmentation of Polyps in Colonoscopy
• NeuBTF: Neural fields for BTF encoding and transfer
• Material-Response-Informed DeepONet and its Application to Polycrystal Stress-strain Prediction in Crystal Plasticity
• Semantic Segmentation of Anaemic RBCs Using Multilevel Deep Convolutional Encoder-Decoder Network
• Speckle-correlation imaging through a kaleidoscopic multimode fiber
• Parser-Free Virtual Try-on via Distilling Appearance Flows
• Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images
• Foreground Removal of CO Intensity Mapping Using Deep Learning
• Heuristical Comparison of Vision Transformers Against Convolutional Neural Networks for Semantic Segmentation on Remote Sensing Imagery
• More than Encoder: Introducing Transformer Decoder to Upsample
• FIBA: Frequency-Injection based Backdoor Attack in Medical Image Analysis
• Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images
• Exploration of Convolutional Neural Network Architectures for Large Region Map Automation
• Disentangled Cycle Consistency for Highly-realistic Virtual Try-On
• Boosting Few-Shot Semantic Segmentation Via Segment Anything Model
• Land Cover Semantic Segmentation Using ResUNet
• Learning Global and Local Features of Normal Brain Anatomy for Unsupervised Abnormality Detection
• Geographical Knowledge-driven Representation Learning for Remote Sensing Images
• Monitoring of Urban Changes with multi-modal Sentinel 1 and 2 Data in Mariupol, Ukraine, in 2022/23
• SA${}^2$Net: Scale-Adaptive Structure-Affinity Transformation for Spine Segmentation from Ultrasound Volume Projection Imaging
• IRSAMap:Towards Large-Scale, High-Resolution Land Cover Map Vectorization
• TinySeg: Model Optimizing Framework for Image Segmentation on Tiny Embedded Systems
• Ensemble Methods for Multi-Organ Segmentation in CT Series
• Looking for change? Roll the Dice and demand Attention
• Lung Segmentation in Chest X-rays with Res-CR-Net
• Learning with less: label-efficient land cover classification at very high spatial resolution using self-supervised deep learning
• CovSegNet: A Multi Encoder-Decoder Architecture for Improved Lesion Segmentation of COVID-19 Chest CT Scans
• Task-Oriented Low-Dose CT Image Denoising
• CXR-Net: An Artificial Intelligence Pipeline for Quick Covid-19 Screening of Chest X-Rays
• MSDU-net: A Multi-Scale Dilated U-net for Blur Detection
• SILOP: An Automated Framework for Semantic Segmentation Using Image Labels Based on Object Perimeters
• PL-Net: Progressive Learning Network for Medical Image Segmentation
• Discover the Mysteries of the Maya: Selected Contributions from the Machine Learning Challenge & The Discovery Challenge Workshop at ECML PKDD 2021
• Analysis of the Compaction Behavior of Textile Reinforcements in Low-Resolution In-Situ CT Scans via Machine-Learning and Descriptor-Based Methods
• DDU-Net: A Domain Decomposition-Based CNN for High-Resolution Image Segmentation on Multiple GPUs
• Image resolution enhancement for advanced semiconductor nodes