Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

Abstract

We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution -- but complete -- output. To this end, we introduce a 3D-Encoder-Predictor Network (3D-EPN) which is composed of 3D convolutional layers. The network is trained to predict and fill in missing data, and operates on an implicit surface representation that encodes both known and unknown space. This allows us to predict global structure in unknown areas at high accuracy. We then correlate these intermediary results with 3D geometry from a shape database at test time. In a final pass, we propose a patch-based 3D shape synthesis method that imposes the 3D geometry from these retrieved shapes as constraints on the coarsely-completed mesh. This synthesis process enables us to reconstruct fine-scale detail and generate high-resolution output while respecting the global mesh structure obtained by the 3D-EPN. Although our 3D-EPN outperforms state-of-the-art completion method, the main contribution in our work lies in the combination of a data-driven shape predictor and analytic 3D shape synthesis. In our results, we show extensive evaluations on a newly-introduced shape completion benchmark for both real-world and synthetic data.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• A Large Dataset of Object Scans
• BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration
• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• VConv-DAE: Deep Volumetric Shape Learning Without Object Labels

Cited by in corpus (36)

• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• SegMap: 3D Segment Mapping using Data-Driven Descriptors
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• PU-Net: Point Cloud Upsampling Network
• High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• PCN: Point Completion Network
• PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks
• OctNetFusion: Learning Depth Fusion from Data
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction
• 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Convolutional Occupancy Networks
• Hierarchical Detail Enhancing Mesh-Based Shape Generation with 3D Generative Adversarial Network
• Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image
• Cycle4Completion: Unpaired Point Cloud Completion using Cycle Transformation with Missing Region Coding
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• A Baseline Approach for AutoImplant: the MICCAI 2020 Cranial Implant Design Challenge
• Refinement of Predicted Missing Parts Enhance Point Cloud Completion
• RevealNet: Seeing Behind Objects in RGB-D Scans
• SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation
• IntrA: 3D Intracranial Aneurysm Dataset for Deep Learning
• Style-based Point Generator with Adversarial Rendering for Point Cloud Completion
• COALESCE: Component Assembly by Learning to Synthesize Connections
• SoftPoolNet: Shape Descriptor for Point Cloud Completion and Classification
• AirObject: A Temporally Evolving Graph Embedding for Object Identification
• Parsing Geometry Using Structure-Aware Shape Templates
• GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion
• SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer
• Scan2Mesh: From Unstructured Range Scans to 3D Meshes
• Surface HOF: Surface Reconstruction from a Single Image Using Higher Order Function Networks
• View-Guided Point Cloud Completion
• InSphereNet: a Concise Representation and Classification Method for 3D Object