3D Shape Induction from 2D Views of Multiple Objects

Abstract

In this paper we investigate the problem of inducing a distribution over three-dimensional structures given two-dimensional views of multiple objects taken from unknown viewpoints. Our approach called "projective generative adversarial networks" (PrGANs) trains a deep generative model of 3D shapes whose projections match the distributions of the input 2D views. The addition of a projection module allows us to infer the underlying 3D shape distribution without using any 3D, viewpoint information, or annotation during the learning phase. We show that our approach produces 3D shapes of comparable quality to GANs trained on 3D data for a number of shape categories including chairs, airplanes, and cars. Experiments also show that the disentangled representation of 2D shapes into geometry and viewpoint leads to a good generative model of 2D shapes. The key advantage is that our model allows us to predict 3D, viewpoint, and generate novel views from an input image in a completely unsupervised manner.

Submitted to CVPR 2017

References in corpus (13)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Fully Convolutional Networks for Semantic Segmentation
• Deep Convolutional Inverse Graphics Network
• A Kernel Method for the Two-Sample Problem
• Deep Shading: Convolutional Neural Networks for Screen-Space Shading
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Unsupervised Learning of 3D Structure from Images
• 3D ShapeNets: A Deep Representation for Volumetric Shapes
• View Synthesis by Appearance Flow
• Pose Induction for Novel Object Categories

Cited by in corpus (10)

• Unsupervised Learning of Depth and Ego-Motion from Video
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
• Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations
• ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
• Unsupervised 3D Shape Learning from Image Collections in the Wild
• End-to-end 3D shape inverse rendering of different classes of objects from a single input image
• 3D Surface Reconstruction by Pointillism