Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

NIPS 2016. The first two authors contributed equally to this work

Cited by in corpus (84)

• Towards Principled Methods for Training Generative Adversarial Networks
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Mode Regularized Generative Adversarial Networks
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Deep Learning for Lung Cancer Detection: Tackling the Kaggle Data Science Bowl 2017 Challenge
• PolyGen: An Autoregressive Generative Model of 3D Meshes
• Improved Adversarial Systems for 3D Object Generation and Reconstruction
• Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
• MeshGAN: Non-linear 3D Morphable Models of Faces
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• PF-Net: Point Fractal Network for 3D Point Cloud Completion
• Adversarial Generation of Training Examples: Applications to Moving Vehicle License Plate Recognition
• Dominant Set Clustering and Pooling for Multi-View 3D Object Recognition
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• Self-Supervised Deep Learning on Point Clouds by Reconstructing Space
• 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks
• Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks
• SurfNet: Generating 3D shape surfaces using deep residual networks
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• 3D-A-Nets: 3D Deep Dense Descriptor for Volumetric Shapes with Adversarial Networks
• Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
• RED: A ReRAM-based Deconvolution Accelerator
• Interactive 3D Modeling with a Generative Adversarial Network
• Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction
• GRASS: Generative Recursive Autoencoders for Shape Structures
• 3D Shape Induction from 2D Views of Multiple Objects
• Message Passing Multi-Agent GANs
• NeuralSampler: Euclidean Point Cloud Auto-Encoder and Sampler
• Learning to Synthesize a 4D RGBD Light Field from a Single Image
• Distributional Adversarial Networks
• How to Hallucinate Functional Proteins
• Conditional Single-view Shape Generation for Multi-view Stereo Reconstruction
• Hierarchical Detail Enhancing Mesh-Based Shape Generation with 3D Generative Adversarial Network
• TripletGAN: Training Generative Model with Triplet Loss
• A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images
• 3DViewGraph: Learning Global Features for 3D Shapes from A Graph of Unordered Views with Attention
• Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
• Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
• Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views
• MRI to CT Translation with GANs
• PixelNN: Example-based Image Synthesis
• Synthesizing facial photometries and corresponding geometries using generative adversarial networks
• Cerberus: A Multi-headed Derenderer
• FMRI data augmentation via synthesis
• LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks
• Hybrid VAE: Improving Deep Generative Models using Partial Observations
• Shape Generation using Spatially Partitioned Point Clouds
• The Automated Inspection of Opaque Liquid Vaccines
• Feeding the zombies: Synthesizing brain volumes using a 3D progressive growing GAN
• Mesh-based Autoencoders for Localized Deformation Component Analysis
• Compressing Representations for Embedded Deep Learning
• Latent feature disentanglement for 3D meshes
• P${}^2$GNet: Pose-Guided Point Cloud Generating Networks for 6-DoF Object Pose Estimation
• Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes
• Latent Variable Modeling for Generative Concept Representations and Deep Generative Models
• Volumetric Convolution: Automatic Representation Learning in Unit Ball
• 3D Semantic Scene Completion from a Single Depth Image using Adversarial Training
• Semantic Photometric Bundle Adjustment on Natural Sequences
• GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion
• Towards Grounding Conceptual Spaces in Neural Representations
• Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
• Learning Generative Models of Structured Signals from Their Superposition Using GANs with Application to Denoising and Demixing
• Learning Embedding of 3D models with Quadric Loss
• Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network
• Deep Structure for end-to-end inverse rendering
• RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials
• Depth Based Semantic Scene Completion with Position Importance Aware Loss
• StructEdit: Learning Structural Shape Variations
• Inferring 3D Shapes from Image Collections using Adversarial Networks
• Learning Quadrangulated Patches For 3D Shape Processing
• Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networks
• 3D Organ Shape Reconstruction from Topogram Images
• LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
• Assembling Semantically-Disentangled Representations for Predictive-Generative Models via Adaptation from Synthetic Domain
• IVE-GAN: Invariant Encoding Generative Adversarial Networks
• Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency
• Towards a Uniform Architecture for the Efficient Implementation of 2D and 3D Deconvolutional Neural Networks on FPGAs
• Deep Generative Models: Deterministic Prediction with an Application in Inverse Rendering
• MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks
• Authoring image decompositions with generative models
• What Do Single-view 3D Reconstruction Networks Learn?
• Design, Analysis and Application of A Volumetric Convolutional Neural Network
• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images