3D ShapeNets: A Deep Representation for Volumetric Shapes

Abstract

3D shape is a crucial but heavily underutilized cue in today's computer vision systems, mostly due to the lack of a good generic shape representation. With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft Kinect), it is becoming increasingly important to have a powerful 3D shape representation in the loop. Apart from category recognition, recovering full 3D shapes from view-based 2.5D depth maps is also a critical part of visual understanding. To this end, we propose to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically. It naturally supports joint object recognition and shape completion from 2.5D depth maps, and it enables active object recognition through view planning. To train our 3D deep learning model, we construct ModelNet -- a large-scale 3D CAD model dataset. Extensive experiments show that our 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.

arXiv commentto be appeared in CVPR 2015

References in corpus (1)

• Nonmyopic View Planning for Active Object Detection

Cited by in corpus (29)

• Cross-view Semantic Segmentation for Sensing Surroundings
• PointCNN: Convolution On $\mathcal{X}$-Transformed Points
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
• Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling
• Patch-based Progressive 3D Point Set Upsampling
• OctNet: Learning Deep 3D Representations at High Resolutions
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• OctNetFusion: Learning Depth Fusion from Data
• 3D-A-Nets: 3D Deep Dense Descriptor for Volumetric Shapes with Adversarial Networks
• 3D Shape Induction from 2D Views of Multiple Objects
• NeuralSampler: Euclidean Point Cloud Auto-Encoder and Sampler
• Wide and deep volumetric residual networks for volumetric image classification
• Iterative Distance-Aware Similarity Matrix Convolution with Mutual-Supervised Point Elimination for Efficient Point Cloud Registration
• Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction
• Poisoning MorphNet for Clean-Label Backdoor Attack to Point Clouds
• Learning 3D-3D Correspondences for One-shot Partial-to-partial Registration
• SampleAhead: Online Classifier-Sampler Communication for Learning from Synthesized Data
• Point Clouds Learning with Attention-based Graph Convolution Networks
• SurfConv: Bridging 3D and 2D Convolution for RGBD Images
• A Unified Game-Theoretic Interpretation of Adversarial Robustness
• Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
• Shape from Shading through Shape Evolution
• URSA: A Neural Network for Unordered Point Clouds Using Constellations
• Learning Kinematic Descriptions using SPARE: Simulated and Physical ARticulated Extendable dataset
• Automatic generation of object shapes with desired functionalities
• 3DMotion-Net: Learning Continuous Flow Function for 3D Motion Prediction
• Quantifying and Learning Linear Symmetry-Based Disentanglement
• Surrogate Modelling for Injection Molding Processes using Machine Learning