RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Abstract

Traditional computer graphics rendering pipeline is designed for procedurally generating 2D quality images from 3D shapes with high performance. The non-differentiability due to discrete operations such as visibility computation makes it hard to explicitly correlate rendering parameters and the resulting image, posing a significant challenge for inverse rendering tasks. Recent work on differentiable rendering achieves differentiability either by designing surrogate gradients for non-differentiable operations or via an approximate but differentiable renderer. These methods, however, are still limited when it comes to handling occlusion, and restricted to particular rendering effects. We present RenderNet, a differentiable rendering convolutional network with a novel projection unit that can render 2D images from 3D shapes. Spatial occlusion and shading calculation are automatically encoded in the network. Our experiments show that RenderNet can successfully learn to implement different shaders, and can be used in inverse rendering tasks to estimate shape, pose, lighting and texture from a single image.

arXiv comment14 pages, 9 figures

References in corpus (8)

• Deep Convolutional Inverse Graphics Network
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• Neural Inverse Rendering for General Reflectance Photometric Stereo
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• 3D-Assisted Image Feature Synthesis for Novel Views of an Object

Cited by in corpus (11)

• Differentiable Surface Splatting for Point-based Geometry Processing
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Differentiable Direct Volume Rendering
• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Stylized Neural Painting