3D-GMNet: Single-View 3D Shape Recovery as A Gaussian Mixture

Abstract

In this paper, we introduce 3D-GMNet, a deep neural network for 3D object shape reconstruction from a single image. As the name suggests, 3D-GMNet recovers 3D shape as a Gaussian mixture. In contrast to voxels, point clouds, or meshes, a Gaussian mixture representation provides an analytical expression with a small memory footprint while accurately representing the target 3D shape. At the same time, it offers a number of additional advantages including instant pose estimation and controllable level-of-detail reconstruction, while also enabling interpretation as a point cloud, volume, and a mesh model. We train 3D-GMNet end-to-end with single input images and corresponding 3D models by introducing two novel loss functions, a 3D Gaussian mixture loss and a 2D multi-view loss, which collectively enable accurate shape reconstruction as kernel density estimation. We thoroughly evaluate the effectiveness of 3D-GMNet with synthetic and real images of objects. The results show accurate reconstruction with a compact representation that also realizes novel applications of single-image 3D reconstruction.

arXiv commentBMVC 2020

References in corpus (5)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling