Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Abstract

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/

Accepted to NeurIPS 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• 3D-Aware Scene Manipulation via Inverse Graphics
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects
• Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• Unsupervised 3D Shape Learning from Image Collections in the Wild

Cited by in corpus (48)

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
• Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs
• Category-Level Metric Scale Object Shape and Pose Estimation
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories
• CPS++: Improving Class-level 6D Pose and Shape Estimation From Monocular Images With Self-Supervised Learning
• Artemis: Articulated Neural Pets with Appearance and Motion synthesis
• Convolutional Generation of Textured 3D Meshes
• 3D Shape Reconstruction from Vision and Touch
• PVA: Pixel-aligned Volumetric Avatars
• ADOP: Approximate Differentiable One-Pixel Point Rendering
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• Differentiable Transient Rendering
• Implicit Mesh Reconstruction from Unannotated Image Collections
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
• ARCH: Animatable Reconstruction of Clothed Humans
• SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
• TM-NET: Deep Generative Networks for Textured Meshes
• Differentiable Rendering with Perturbed Optimizers
• Shape from Projections via Differentiable Forward Projector for Computed Tomography
• Revisiting Latent-Space Interpolation via a Quantitative Evaluation Framework
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency
• Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation
• Learning with Algorithmic Supervision via Continuous Relaxations
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
• PG-NeuS: Robust and Efficient Point Guidance for Multi-View Neural Surface Reconstruction
• GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
• Canonical 3D Deformer Maps: Unifying parametric and non-parametric methods for dense weakly-supervised category reconstruction
• Towards Differentiable Rendering for Sidescan Sonar Imagery
• Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
• End-to-end Generative Zero-shot Learning via Few-shot Learning
• Interactive Annotation of 3D Object Geometry using 2D Scribbles
• Texture Generation Using A Graph Generative Adversarial Network And Differentiable Rendering
• De-rendering the World's Revolutionary Artefacts
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction
• ContourRender: Detecting Arbitrary Contour Shape For Instance Segmentation In One Pass
• GAMesh: Guided and Augmented Meshing for Deep Point Networks