3D Guided Weakly Supervised Semantic Segmentation
arXiv:2012.00242
Abstract
Pixel-wise clean annotation is necessary for fully-supervised semantic segmentation, which is laborious and expensive to obtain. In this paper, we propose a weakly supervised 2D semantic segmentation model by incorporating sparse bounding box labels with available 3D information, which is much easier to obtain with advanced sensors. We manually labeled a subset of the 2D-3D Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D inference module to generate accurate pixel-wise segment proposal masks. Guided by 3D information, we first generate a point cloud of objects and calculate objectness probability score for each point. Then we project the point cloud with objectness probabilities back to 2D images followed by a refinement step to obtain segment proposals, which are treated as pseudo labels to train a semantic segmentation network. Our method works in a recursive manner to gradually refine the above-mentioned segment proposals. Extensive experimental results on the 2D-3D-S dataset show that the proposed method can generate accurate segment proposals when bounding box labels are available on only a small subset of training images. Performance comparison with recent state-of-the-art methods further illustrates the effectiveness of our method.
References in corpus (9)
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- ParseNet: Looking Wider to See Better
- Joint 2D-3D-Semantic Data for Indoor Scene Understanding
- Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation
- ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
- OCNet: Object Context Network for Scene Parsing
- Matterport3D: Learning from RGB-D Data in Indoor Environments