Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

Abstract

The long-coveted task of reconstructing 3D geometry from images is still a standing problem. In this paper, we build on the power of neural networks and introduce Pix2Vex, a network trained to convert camera-captured images into 3D geometry. We present a novel differentiable renderer ($DR$) as a forward validation means during training. Our key insight is that $DR$s produce images of a particular appearance, different from typical input images. Hence, we propose adding an image-to-image translation component, converting between these rendering styles. This translation closes the training loop, while allowing to use minimal supervision only, without needing any 3D model as ground truth. Unlike state-of-the-art methods, our $DR$ is $C^{\infty }$ smooth and thus does not display any discontinuities at occlusions or dis-occlusions. Through our novel training scheme, our network can train on different types of images, where previous work can typically only train on images of a similar appearance to those rendered by a $DR$.

References in corpus (11)

• Synthesizing Robust Adversarial Examples
• Learning Representations and Generative Models for 3D Point Clouds
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• Inverse Transport Networks
• 3D Shape Induction from 2D Views of Multiple Objects
• First Experiments with Neural Translation of Informal to Formal Mathematics
• CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

Cited by in corpus (14)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Differentiable Surface Splatting for Point-based Geometry Processing
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• Differentiable Rendering: A Survey
• Differentiable Direct Volume Rendering
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
• A Neural Rendering Framework for Free-Viewpoint Relighting
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• Learning with Algorithmic Supervision via Continuous Relaxations
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images
• Neural View-Interpolation for Sparse Light Field Video
• AlgoNet: $C^{\infty }$ Smooth Algorithmic Neural Networks