Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks

Abstract

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Long-term Recurrent Convolutional Network (LRCN). The 3D-ED-GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neural network architecture to minimize GPU memory usage and incorporates an Encoder-Decoder pair into a Long Short-term Memory Network. By handling the 3D model as a sequence of 2D slices, LRCN transforms a coarse 3D shape into a more complete and higher resolution volume. While 3D-ED-GAN captures global contextual structure of the 3D shape, LRCN localizes the fine-grained details. Experimental results on both real-world and synthetic data show reconstructions from corrupted models result in complete and high-resolution 3D objects.

References in corpus (7)

• Auto-Encoding Variational Bayes
• Recurrent Neural Network Regularization
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• VConv-DAE: Deep Volumetric Shape Learning Without Object Labels

Cited by in corpus (8)

• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Peeking Behind Objects: Layered Depth Prediction from a Single Image
• PF-Net: Point Fractal Network for 3D Point Cloud Completion
• Video Inpainting by Jointly Learning Temporal Structure and Spatial Details
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• KTNet: Knowledge Transfer for Unpaired 3D Shape Completion
• GANs-NQM: A Generative Adversarial Networks based No Reference Quality Assessment Metric for RGB-D Synthesized Views
• Boosting Image Outpainting with Semantic Layout Prediction