Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Abstract

Rendering is the process of generating 2D images from 3D assets, simulated in a virtual environment, typically with a graphics pipeline. By inverting such renderer, one can think of a learning approach to predict a 3D shape from an input image. However, standard rendering pipelines involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence suitable for learning. We present the first non-parametric and truly differentiable rasterizer based on silhouettes. Our method enables unsupervised learning for high-quality 3D mesh reconstruction from a single image. We call our framework `soft rasterizer' as it provides an accurate soft approximation of the standard rasterizer. The key idea is to fuse the probabilistic contributions of all mesh triangles with respect to the rendered pixels. When combined with a mesh generator in a deep neural network, our soft rasterizer is able to generate an approximated silhouette of the generated polygon mesh in the forward pass. The rendering loss is back-propagated to supervise the mesh generation without the need of 3D training data. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art unsupervised techniques, both quantitatively and qualitatively. We also show that our soft rasterizer can achieve comparable results to the cutting-edge supervised learning method and in various cases even better ones, especially for real-world data.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• Learning Category-Specific Mesh Reconstruction from Image Collections
• Nonlinear 3D Face Morphable Model
• Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation
• SiCloPe: Silhouette-Based Clothed People
• HairNet: Single-View Hair Reconstruction using Convolutional Neural Networks

Cited by in corpus (26)

• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• Accelerating 3D Deep Learning with PyTorch3D
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• Towards Universal Physical Attacks On Cascaded Camera-Lidar 3D Object Detection Models
• Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• Physically Realizable Adversarial Examples for LiDAR Object Detection
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images
• Neural Sparse Voxel Fields
• Neural Radiance Flow for 4D View Synthesis and Video Processing
• Shape from Projections via Differentiable Forward Projector for Computed Tomography
• Self-supervised Learning of 3D Objects from Natural Images
• E3D: Event-Based 3D Shape Reconstruction
• Object-Centric Image Generation with Factored Depths, Locations, and Appearances
• Recognizing Scenes from Novel Viewpoints
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Towards Differentiable Rendering for Sidescan Sonar Imagery
• Generating Novel Glyph without Human Data by Learning to Communicate
• Deep Feedback Inverse Problem Solver
• Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency
• Neural Implicit 3D Shapes from Single Images with Spatial Patterns
• Inverting Generative Adversarial Renderer for Face Reconstruction
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction
• Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction