Visual Object Networks: Image Generation with Disentangled 3D Representation
arXiv:1812.02725
Abstract
Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors---shape, viewpoint, and texture---and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.
arXiv commentNeurIPS 2018. Code: https://github.com/junyanz/VON Website: http://von.csail.mit.edu/
References in corpus (3)
Cited by in corpus (21)
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
- BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
- CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
- Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
- Convolutional Generation of Textured 3D Meshes
- PVA: Pixel-aligned Volumetric Avatars
- Towards Realistic 3D Embedding via View Alignment
- SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
- TM-NET: Deep Generative Networks for Textured Meshes
- Transformable Bottleneck Networks
- Self-supervised Single-view 3D Reconstruction via Semantic Consistency
- Multimodal Shape Completion via Conditional Generative Adversarial Networks
- Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence
- Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
- Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images
- Hard Example Generation by Texture Synthesis for Cross-domain Shape Similarity Learning
- Learning Compositional Representation for 4D Captures with Neural ODE
- Fine-grained Image-to-Image Transformation towards Visual Recognition
- Disentangling semantic features of macromolecules in Cryo-Electron Tomography
- Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images
- Novel View Synthesis via Depth-guided Skip Connections