Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning

Abstract

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.

This is a substantially revised version of previous submission: arXiv:1901.05567

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
• Learning Category-Specific Mesh Reconstruction from Image Collections

Cited by in corpus (45)

• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• pixelNeRF: Neural Radiance Fields from One or Few Images
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• CPS++: Improving Class-level 6D Pose and Shape Estimation From Monocular Images With Self-Supervised Learning
• PVA: Pixel-aligned Volumetric Avatars
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks
• Implicit Mesh Reconstruction from Unannotated Image Collections
• DOVE: Learning Deformable 3D Objects by Watching Videos
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images
• ARCH: Animatable Reconstruction of Clothed Humans
• TM-NET: Deep Generative Networks for Textured Meshes
• Differentiable Rendering with Perturbed Optimizers
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Forward Prediction for Physical Reasoning
• Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
• Interpolating Points on a Non-Uniform Grid using a Mixture of Gaussians
• Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Deep 3D-to-2D Watermarking: Embedding Messages in 3D Meshes and Extracting Them from 2D Renderings
• Imagine That! Leveraging Emergent Affordances for 3D Tool Synthesis
• MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering
• SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• Deep Optimized Priors for 3D Shape Modeling and Reconstruction
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images
• Stylized Neural Painting
• Coherent Reconstruction of Multiple Humans from a Single Image
• Neutral Face Game Character Auto-Creation via PokerFace-GAN
• End-to-End Optimization of Scene Layout
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction
• ContourRender: Detecting Arbitrary Contour Shape For Instance Segmentation In One Pass
• Physical world assistive signals for deep neural network classifiers -- neither defense nor attack
• GAMesh: Guided and Augmented Meshing for Deep Point Networks
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects