LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Abstract

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.

References in corpus (6)

• Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Playing for Data: Ground Truth from Computer Games
• Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge
• A Dataset for Improved RGBD-based Object Detection and Pose Estimation for Warehouse Pick-and-Place

Cited by in corpus (4)

• Block Annotation: Better Image Annotation for Semantic Segmentation with Sub-Image Decomposition
• A Dynamical Perspective on Point Cloud Registration
• Pose Estimation for Texture-less Shiny Objects in a Single RGB Image Using Synthetic Training Data
• S4-Net: Geometry-Consistent Semi-Supervised Semantic Segmentation