DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Abstract

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward passes of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backwards to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises.

arXiv commentCamera-ready version to appear in CVPR 2020. Project page: http://b1ueber2y.me/projects/DIST-Renderer

References in corpus (14)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
• A Large Dataset of Object Scans
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• End-to-end Recovery of Human Shape and Pose
• Deep Level Sets: Implicit Surface Representations for 3D Shape Inference
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• OctNet: Learning Deep 3D Representations at High Resolutions
• Conditional Single-view Shape Generation for Multi-view Stereo Reconstruction
• Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction

Cited by in corpus (7)

• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
• SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images
• Neural Sparse Voxel Fields
• Monocular Real-Time Volumetric Performance Capture
• MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
• Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry