Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Data
arXiv:2301.07283 · doi:10.1109/CRV60082.2023.00026
Abstract
Reducing the quantity of annotations required for supervised training is vital when labels are scarce and costly. This reduction is particularly important for semantic segmentation tasks involving 3D datasets, which are often significantly smaller and more challenging to annotate than their image-based counterparts. Self-supervised pre-training on unlabelled data is one way to reduce the amount of manual annotations needed. Previous work has focused on pre-training with point clouds exclusively. While useful, this approach often requires two or more registered views. In the present work, we combine image and point cloud modalities by first learning self-supervised image features and then using these features to train a 3D model. By incorporating image data, which is often included in many 3D datasets, our pre-training method only requires a single scan of a scene and can be applied to cases where localization information is unavailable. We demonstrate that our pre-training approach, despite using single scans, achieves comparable performance to other multi-scan, point cloud-only methods.
arXiv commentIn Proceedings of the Conference on Robots and Vision (CRV'23), Montreal, Canada, Jun. 6-8, 2023. arXiv admin note: substantial text overlap with arXiv:2211.11801
References in corpus (9)
- A Simple Framework for Contrastive Learning of Visual Representations
- Representation Learning with Contrastive Predictive Coding
- Bootstrap your own latent: A new approach to self-supervised Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Contrastive Representation Learning: A Framework and Review
- Prototypical Contrastive Learning of Unsupervised Representations
- Hard Negative Mixing for Contrastive Learning
- P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
- Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining