Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network

Abstract

Object detection and semantic segmentation are two main themes in object retrieval from high-resolution remote sensing images, which have recently achieved remarkable performance by surfing the wave of deep learning and, more notably, convolutional neural networks (CNNs). In this paper, we are interested in a novel, more challenging problem of vehicle instance segmentation, which entails identifying, at a pixel-level, where the vehicles appear as well as associating each pixel with a physical instance of a vehicle. In contrast, vehicle detection and semantic segmentation each only concern one of the two. We propose to tackle this problem with a semantic boundary-aware multi-task learning network. More specifically, we utilize the philosophy of residual learning (ResNet) to construct a fully convolutional network that is capable of harnessing multi-level contextual feature representations learned from different residual blocks. We theoretically analyze and discuss why residual networks can produce better probability maps for pixel-wise segmentation tasks. Then, based on this network architecture, we propose a unified multi-task learning network that can simultaneously learn two complementary tasks, namely, segmenting vehicle regions and detecting semantic boundaries. The latter subproblem is helpful for differentiating closely spaced vehicles, which are usually not correctly separated into instances. Currently, datasets with pixel-wise annotation for vehicle extraction are ISPRS dataset and IEEE GRSS DFC2015 dataset over Zeebrugge, which specializes in semantic segmentation. Therefore, we built a new, more challenging dataset for vehicle instance segmentation, called the Busy Parking Lot UAV Video dataset, and we make our dataset available at http://www.sipeo.bgu.tum.de/download so that it can be used to benchmark future vehicle instance segmentation algorithms.

Preprint of a paper accepted by IEEE Transactions On Geoscience and Remote Sensing

References in corpus (3)

• Deep learning in remote sensing: a review
• Conditional Random Fields as Recurrent Neural Networks
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks

Cited by in corpus (13)

• A Survey of Deep Learning-based Object Detection
• Landslide4Sense: Reference Benchmark Data and Deep Learning Models for Landslide Detection
• HED-UNet: Combined Segmentation and Edge Detection for Monitoring the Antarctic Coastline
• Recurrently Exploring Class-wise Attention in A Hybrid Convolutional and Bidirectional LSTM Network for Multi-label Aerial Image Classification
• Multiple instance dense connected convolution neural network for aerial image scene classification
• R${}^3$-Net: A Deep Network for Multi-oriented Vehicle Detection in Aerial Images and Videos
• Relation Network for Multi-label Aerial Image Classification
• Aerial Images Processing for Car Detection using Convolutional Neural Networks: Comparison between Faster R-CNN and YoloV3
• Panoptic Segmentation Meets Remote Sensing
• All Grains, One Scheme (AGOS): Learning Multi-grain Instance Representation for Aerial Scene Classification
• Bounding Box-Free Instance Segmentation Using Semi-Supervised Learning for Generating a City-Scale Vehicle Dataset
• Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery
• FuTH-Net: Fusing Temporal Relations and Holistic Features for Aerial Video Classification