RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

Abstract

Semantic scene understanding from point clouds is particularly challenging as the points reflect only a sparse set of the underlying 3D geometry. Previous works often convert point cloud into regular grids (e.g. voxels or bird-eye view images), and resort to grid-based convolutions for scene understanding. In this work, we introduce RfD-Net that jointly detects and reconstructs dense object surfaces directly from raw point clouds. Instead of representing scenes with regular grids, our method leverages the sparsity of point cloud data and focuses on predicting shapes that are recognized with high objectness. With this design, we decouple the instance reconstruction into global object localization and local shape prediction. It not only eases the difficulty of learning 2-D manifold surfaces from sparse 3D space, the point clouds in each object proposal convey shape details that support implicit function learning to reconstruct any high-resolution surfaces. Our experiments indicate that instance detection and reconstruction present complementary effects, where the shape prediction head shows consistent effects on improving object detection with modern 3D proposal network backbones. The qualitative and quantitative evaluations further demonstrate that our approach consistently outperforms the state-of-the-arts and improves over 11 of mesh IoU in object reconstruction.

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Adversarially Learned Inference
• Learning Representations and Generative Models for 3D Point Clouds
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• Skeleton-bridged Point Completion: From Global Inference to Local Adjustment
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• FroDO: From Detections to 3D Objects
• SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
• RevealNet: Seeing Behind Objects in RGB-D Scans
• Attention-based Multi-modal Fusion Network for Semantic Scene Completion
• SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans

Cited by in corpus (1)

• TransformerFusion: Monocular RGB Scene Reconstruction using Transformers