Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

NIPS 2016. The first two authors contributed equally to this work

Cited by in corpus (168)

• Towards Principled Methods for Training Generative Adversarial Networks
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Mode Regularized Generative Adversarial Networks
• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• Differentiable Rendering: A Survey
• Accelerating 3D Deep Learning with PyTorch3D
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• Neural Marching Cubes
• Deep Learning for Lung Cancer Detection: Tackling the Kaggle Data Science Bowl 2017 Challenge
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• PolyGen: An Autoregressive Generative Model of 3D Meshes
• Improved Adversarial Systems for 3D Object Generation and Reconstruction
• Category-Level Metric Scale Object Shape and Pose Estimation
• Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
• Unsupervised Generative 3D Shape Learning from Natural Images
• MeshGAN: Non-linear 3D Morphable Models of Faces
• Self-Supervised Few-Shot Learning on Point Clouds
• Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image
• PF-Net: Point Fractal Network for 3D Point Cloud Completion
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design
• Dominant Set Clustering and Pooling for Multi-View 3D Object Recognition
• Adversarial Generation of Training Examples: Applications to Moving Vehicle License Plate Recognition
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• Self-Supervised Deep Learning on Point Clouds by Reconstructing Space
• Neural Unsigned Distance Fields for Implicit Function Learning
• 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks
• Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks
• On the Benefit of Adversarial Training for Monocular Depth Estimation
• SurfNet: Generating 3D shape surfaces using deep residual networks
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• 3D-A-Nets: 3D Deep Dense Descriptor for Volumetric Shapes with Adversarial Networks
• Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
• Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
• RED: A ReRAM-based Deconvolution Accelerator
• SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation
• Deep Generative Model for Efficient 3D Airfoil Parameterization and Generation
• SoftPool++: An Encoder-Decoder Network for Point Cloud Completion
• Interactive 3D Modeling with a Generative Adversarial Network
• Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction
• GRASS: Generative Recursive Autoencoders for Shape Structures
• 3D Shape Induction from 2D Views of Multiple Objects
• 3DGAUnet: 3D generative adversarial networks with a 3D U-Net based generator to achieve the accurate and effective synthesis of clinical tumor image data for pancreatic cancer
• Zero-Shot Text-Guided Object Generation with Dream Fields
• Message Passing Multi-Agent GANs
• Fit2Form: 3D Generative Model for Robot Gripper Form Design
• NeuralSampler: Euclidean Point Cloud Auto-Encoder and Sampler
• Learning to Synthesize a 4D RGBD Light Field from a Single Image
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• Distributional Adversarial Networks
• Self-supervised Modal and View Invariant Feature Learning
• How to Hallucinate Functional Proteins
• Learning Compositional Radiance Fields of Dynamic Human Heads
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning
• Conditional Single-view Shape Generation for Multi-view Stereo Reconstruction
• A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time
• Learning Gradient Fields for Shape Generation
• A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images
• Hierarchical Detail Enhancing Mesh-Based Shape Generation with 3D Generative Adversarial Network
• TripletGAN: Training Generative Model with Triplet Loss
• 3DViewGraph: Learning Global Features for 3D Shapes from A Graph of Unordered Views with Attention
• Deep Implicit Moving Least-Squares Functions for 3D Reconstruction
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
• Progressive Point Cloud Deconvolution Generation Network
• Extreme Relative Pose Estimation for RGB-D Scans via Scene Completion
• Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views
• MRI to CT Translation with GANs
• Augmenting Implicit Neural Shape Representations with Explicit Deformation Fields
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds
• Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation
• Synthesizing facial photometries and corresponding geometries using generative adversarial networks
• FMRI data augmentation via synthesis
• Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
• Hierarchical Point Cloud Encoding and Decoding with Lightweight Self-Attention based Model
• Cerberus: A Multi-headed Derenderer
• Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
• DeepPoint: A Deep Learning Model for 3D Reconstruction in Point Clouds via mmWave Radar
• Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve
• Fed-Sim: Federated Simulation for Medical Imaging
• PixelNN: Example-based Image Synthesis
• Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences
• TetraTSDF: 3D human reconstruction from a single image with a tetrahedral outer shell
• Shape As Points: A Differentiable Poisson Solver
• The Automated Inspection of Opaque Liquid Vaccines
• A Simple and Scalable Shape Representation for 3D Reconstruction
• Feeding the zombies: Synthesizing brain volumes using a 3D progressive growing GAN
• Unsupervised Partial Point Set Registration via Joint Shape Completion and Registration
• Hybrid VAE: Improving Deep Generative Models using Partial Observations
• HyperFlow: Representing 3D Objects as Surfaces
• LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks
• Shape Generation using Spatially Partitioned Point Clouds
• SALD: Sign Agnostic Learning with Derivatives
• Latent feature disentanglement for 3D meshes
• Mesh-based Autoencoders for Localized Deformation Component Analysis
• Compressing Representations for Embedded Deep Learning
• DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces
• Data Augmentation for Enhancing EEG-based Emotion Recognition with Deep Generative Models
• Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion
• CodeNeRF: Disentangled Neural Radiance Fields for Object Categories
• Dynamic Plane Convolutional Occupancy Networks
• GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion
• 3D Semantic Scene Completion from a Single Depth Image using Adversarial Training
• P${}^2$GNet: Pose-Guided Point Cloud Generating Networks for 6-DoF Object Pose Estimation
• Latent Variable Modeling for Generative Concept Representations and Deep Generative Models
• Volumetric Convolution: Automatic Representation Learning in Unit Ball
• Semantic Photometric Bundle Adjustment on Natural Sequences
• Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes
• Towards Grounding Conceptual Spaces in Neural Representations
• Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network
• Depth Based Semantic Scene Completion with Position Importance Aware Loss
• 3DMaterialGAN: Learning 3D Shape Representation from Latent Space for Materials Science Applications
• Learning Generative Models of Structured Signals from Their Superposition Using GANs with Application to Denoising and Demixing
• Deriving Visual Semantics from Spatial Context: An Adaptation of LSA and Word2Vec to generate Object and Scene Embeddings from Images
• RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials
• Deep Structure for end-to-end inverse rendering
• Learning Embedding of 3D models with Quadric Loss
• Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
• DmifNet:3D Shape Reconstruction Based on Dynamic Multi-Branch Information Fusion
• Diverse Plausible Shape Completions from Ambiguous Depth Images
• HyperCube: Implicit Field Representations of Voxelized 3D Models
• DECOR-GAN: 3D Shape Detailization by Conditional Refinement
• Staying in Shape: Learning Invariant Shape Representations using Contrastive Learning
• StructEdit: Learning Structural Shape Variations
• 3D Brain Reconstruction by Hierarchical Shape-Perception Network from a Single Incomplete Image
• A Point Cloud Generative Model via Tree-Structured Graph Convolutions for 3D Brain Shape Reconstruction
• Learning Quadrangulated Patches For 3D Shape Processing
• Geodesic-HOF: 3D Reconstruction Without Cutting Corners
• Assembling Semantically-Disentangled Representations for Predictive-Generative Models via Adaptation from Synthetic Domain
• DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates
• Learning to Generate 3D Shapes with Generative Cellular Automata
• MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videos
• Learning geometry-image representation for 3D point cloud generation
• A curvature and density-based generative representation of shapes
• 3D Organ Shape Reconstruction from Topogram Images
• LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
• Protecting Anonymous Speech: A Generative Adversarial Network Methodology for Removing Stylistic Indicators in Text
• Synthesizing 3D Shapes from Silhouette Image Collections using Multi-projection Generative Adversarial Networks
• Inferring 3D Shapes from Image Collections using Adversarial Networks
• IVE-GAN: Invariant Encoding Generative Adversarial Networks
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• Image Generation and Recognition (Emotions)
• Single Image 3D Object Estimation with Primitive Graph Networks
• Towards a Uniform Architecture for the Efficient Implementation of 2D and 3D Deconvolutional Neural Networks on FPGAs
• Authoring image decompositions with generative models
• MV-C3D: A Spatial Correlated Multi-View 3D Convolutional Neural Networks
• Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency
• 3D Scattering Tomography by Deep Learning with Architecture Tailored to Cloud Fields
• NeuralQAAD: An Efficient Differentiable Framework for High Resolution Point Cloud Compression
• Hierarchical View Predictor: Unsupervised 3D Global Feature Learning through Hierarchical Prediction among Unordered Views
• Profile to Frontal Face Recognition in the Wild Using Coupled Conditional GAN
• 3D Shape Generation with Grid-based Implicit Functions
• Synthetic Generation of Three-Dimensional Cancer Cell Models from Histopathological Images
• Shelf-Supervised Mesh Prediction in the Wild
• DynOcc: Learning Single-View Depth from Dynamic Occlusion Cues
• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images
• Unsupervised Learning of Depth and Depth-of-Field Effect from Natural Images with Aperture Rendering Generative Adversarial Networks
• ViT-Inception-GAN for Image Colourising
• Design, Analysis and Application of A Volumetric Convolutional Neural Network
• 3D Topology Transformation with Generative Adversarial Networks
• Improving Model Compatibility of Generative Adversarial Networks by Boundary Calibration
• What Do Single-view 3D Reconstruction Networks Learn?
• Latent Space Exploration Using Generative Kernel PCA
• Deep Generative Models: Deterministic Prediction with an Application in Inverse Rendering
• 3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video