Differentiable Rendering: A Survey

Abstract

Deep neural networks (DNNs) have shown remarkable performance improvements on vision-related tasks such as object detection or image segmentation. Despite their success, they generally lack the understanding of 3D objects which form the image, as it is not always possible to collect 3D information about the scene or to easily annotate it. Differentiable rendering is a novel field which allows the gradients of 3D objects to be calculated and propagated through images. It also reduces the requirement of 3D data collection and annotation, while enabling higher success rate in various applications. This paper reviews existing literature and discusses the current state of differentiable rendering, its applications and open research problems.

References in corpus (17)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Differentiable Surface Splatting for Point-based Geometry Processing
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Learning to Infer Implicit Surfaces without 3D Supervision
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• Classification of Point Cloud Scenes with Multiscale Voxel Deep Network
• GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects
• Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
• Cerberus: A Multi-headed Derenderer
• Robust Point Light Source Estimation Using Differentiable Rendering
• STA: Adversarial Attacks on Siamese Trackers

Cited by in corpus (25)

• Differentiable Direct Volume Rendering
• ADOP: Approximate Differentiable One-Pixel Point Rendering
• Differentiable Transient Rendering
• Unsupervised Discovery of Object Radiance Fields
• Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving
• Neural Sparse Voxel Fields
• Smoothing Methods for Automatic Differentiation Across Conditional Branches
• Deep Direct Volume Rendering: Learning Visual Feature Mappings From Exemplary Images
• Data Augmentation for Object Detection via Differentiable Neural Rendering
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing
• Neural Weighted A*: Learning Graph Costs and Heuristics with Differentiable Anytime A*
• Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations
• Neural Image Abstraction Using Long Smoothing B-Splines
• Deep scene-scale material estimation from multi-view indoor captures
• Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering
• Towards Differentiable Rendering for Sidescan Sonar Imagery
• Inverse 3D Microscopy Rendering for Cell Shape Inference with Active Mesh
• Recursive-NeRF: An Efficient and Dynamically Growing NeRF
• Deep Graphics Encoder for Real-Time Video Makeup Synthesis from Example
• Gradients are Not All You Need
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects
• LocaliseBot: Multi-view 3D object localisation with differentiable rendering for robot grasping
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics
• Visual Tomography: Physically Faithful Volumetric Models of Partially Translucent Objects