Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer

Abstract

Semantic annotations are vital for training models for object recognition, semantic segmentation or scene understanding. Unfortunately, pixelwise annotation of images at very large scale is labor-intensive and only little labeled data is available, particularly at instance level and for street scenes. In this paper, we propose to tackle this problem by lifting the semantic instance labeling task from 2D into 3D. Given reconstructions from stereo or laser data, we annotate static 3D scene elements with rough bounding primitives and develop a model which transfers this information into the image domain. We leverage our method to obtain 2D labels for a novel suburban video dataset which we have collected, resulting in 400k semantic and instance image annotations. A comparison of our method to state-of-the-art label transfer baselines reveals that 3D information enables more efficient annotation while at the same time resulting in improved accuracy and time-coherent labels.

10 pages in Conference on Computer Vision and Pattern Recognition (CVPR), 2016

References in corpus (6)

• ADADELTA: An Adaptive Learning Rate Method
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Fully Convolutional Networks for Semantic Segmentation
• Object Detectors Emerge in Deep Scene CNNs
• segDeepM: Exploiting Segmentation and Context in Deep Neural Networks for Object Detection

Cited by in corpus (8)

• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Multi-Spectral Image Synthesis for Crop/Weed Segmentation in Precision Farming
• Towards Collaborative Simultaneous Localization and Mapping: a Survey of the Current Research Landscape
• Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption
• CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation
• No More Discrimination: Cross City Adaptation of Road Scene Segmenters
• Ad-datasets: a meta-collection of data sets for autonomous driving
• MODISSA: a multipurpose platform for the prototypical realization of vehicle-related applications using optical sensors