Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks

Abstract

Reconstructing the 3D mesh of a general object from a single image is now possible thanks to the latest advances of deep learning technologies. However, due to the nontrivial difficulty of generating a feasible mesh structure, the state-of-the-art approaches often simplify the problem by learning the displacements of a template mesh that deforms it to the target surface. Though reconstructing a 3D shape with complex topology can be achieved by deforming multiple mesh patches, it remains difficult to stitch the results to ensure a high meshing quality. In this paper, we present an end-to-end single-view mesh reconstruction framework that is able to generate high-quality meshes with complex topologies from a single genus-0 template mesh. The key to our approach is a novel progressive shaping framework that alternates between mesh deformation and topology modification. While a deformation network predicts the per-vertex translations that reduce the gap between the reconstructed mesh and the ground truth, a novel topology modification network is employed to prune the error-prone faces, enabling the evolution of topology. By iterating over the two procedures, one can progressively modify the mesh topology while achieving higher reconstruction accuracy. Moreover, a boundary refinement network is designed to refine the boundary conditions to further improve the visual quality of the reconstructed mesh. Extensive experiments demonstrate that our approach outperforms the current state-of-the-art methods both qualitatively and quantitatively, especially for the shapes with complex topologies.

arXiv comment10 pages, 11 figures, to be presented at ICCV 2019

References in corpus (6)

• Learning Representations and Generative Models for 3D Point Clouds
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Deep Level Sets: Implicit Surface Representations for 3D Shape Inference
• Deep RBFNet: Point Cloud Feature Learning using Radial Basis Functions

Cited by in corpus (1)

• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art