Learning to Infer Implicit Surfaces without 3D Supervision

Abstract

Recent advances in 3D deep learning have shown that it is possible to train highly effective deep models for 3D shape generation, directly from 2D images. This is particularly interesting since the availability of 3D models is still limited compared to the massive amount of accessible 2D images, which is invaluable for training. The representation of 3D surfaces itself is a key factor for the quality and resolution of the 3D output. While explicit representations, such as point clouds and voxels, can span a wide range of shape variations, their resolutions are often limited. Mesh-based representations are more efficient but are limited by their ability to handle varying topologies. Implicit surfaces, however, can robustly handle complex shapes, topologies, and also provide flexible resolution control. We address the fundamental problem of learning implicit surfaces for shape inference without the need of 3D supervision. Despite their advantages, it remains nontrivial to (1) formulate a differentiable connection between implicit surfaces and their 2D renderings, which is needed for image-based supervision; and (2) ensure precise geometric properties and control, such as local smoothness. In particular, sampling implicit surfaces densely is also known to be a computationally demanding and very slow operation. To this end, we propose a novel ray-based field probing technique for efficient image-to-field supervision, as well as a general geometric regularizer for implicit surfaces, which provides natural shape priors in unconstrained regions. We demonstrate the effectiveness of our framework on the task of single-view image-based 3D shape digitization and show how we outperform state-of-the-art techniques both quantitatively and qualitatively.

Cited by in corpus (38)

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• Differentiable Rendering: A Survey
• pixelNeRF: Neural Radiance Fields from One or Few Images
• Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces
• Differentiable Direct Volume Rendering
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization
• ADOP: Approximate Differentiable One-Pixel Point Rendering
• MeshSDF: Differentiable Iso-Surface Extraction
• Learning 3D Dynamic Scene Representations for Robot Manipulation
• Scalable Surface Reconstruction with Delaunay-Graph Neural Networks
• Learned Initializations for Optimizing Coordinate-Based Neural Representations
• SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images
• Neural Sparse Voxel Fields
• Convolutional Occupancy Networks
• ARCH: Animatable Reconstruction of Clothed Humans
• Neural Radiance Flow for 4D View Synthesis and Video Processing
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency
• Self-supervised Learning of 3D Objects from Natural Images
• Semantic Correspondence via 2D-3D-2D Cycle
• Neural Fields in Visual Computing and Beyond
• SALD: Sign Agnostic Learning with Derivatives
• Editing Conditional Radiance Fields
• Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondence
• Dynamic Plane Convolutional Occupancy Networks
• A Survey on Deep Geometry Learning: From a Representation Perspective
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Deep Optimized Priors for 3D Shape Modeling and Reconstruction
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images
• PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations
• MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
• PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction
• Learning to Reconstruct and Segment 3D Objects
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction
• Point-based Acoustic Scattering for Interactive Sound Propagation via Surface Encoding
• Shelf-Supervised Mesh Prediction in the Wild