Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

Abstract

When working with three-dimensional data, choice of representation is key. We explore voxel-based models, and present evidence for the viability of voxellated representations in applications including shape modeling and object classification. Our key contributions are methods for training voxel-based variational autoencoders, a user interface for exploring the latent space learned by the autoencoder, and a deep convolutional neural network architecture for object classification. We address challenges unique to voxel-based representations, and empirically evaluate our models on the ModelNet benchmark, where we demonstrate a 51.5% relative improvement in the state of the art for object classification.

9 pages, 5 figures, 2 tables

References in corpus (2)

• Auto-Encoding Variational Bayes
• A guide to convolution arithmetic for deep learning

Cited by in corpus (92)

• MeshCNN: A Network with an Edge
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Point-Voxel CNN for Efficient 3D Deep Learning
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Learned Point Cloud Geometry Compression
• SegMap: 3D Segment Mapping using Data-Driven Descriptors
• Dense 3D Object Reconstruction from a Single Depth View
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes
• Weakly-supervised DCNN for RGB-D Object Recognition in Real-World Applications Which Lack Large-scale Annotated Training Data
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
• SO-Net: Self-Organizing Network for Point Cloud Analysis
• SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters
• SPLATNet: Sparse Lattice Networks for Point Cloud Processing
• Category-Level Metric Scale Object Shape and Pose Estimation
• Laplacian2Mesh: Laplacian-Based Mesh Understanding
• Object-Centric Neural Scene Rendering
• RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints
• 3D Point Cloud Classification and Segmentation using 3D Modified Fisher Vector Representation for Convolutional Neural Networks
• Classification of Point Cloud Scenes with Multiscale Voxel Deep Network
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling
• A-CNN: Annularly Convolutional Neural Networks on Point Clouds
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• Triangle-Net: Towards Robustness in Point Cloud Learning
• H-CNN: Spatial Hashing Based CNN for 3D Shape Analysis
• OctNet: Learning Deep 3D Representations at High Resolutions
• PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition
• Interpolated Convolutional Networks for 3D Point Cloud Understanding
• 3D Object Reconstruction from a Single Depth View with Adversarial Learning
• Generating 3D faces using Convolutional Mesh Autoencoders
• Relative Feature Importance
• PS^2-Net: A Locally and Globally Aware Network for Point-Based Semantic Segmentation
• PSNet: Fast Data Structuring for Hierarchical Deep Learning on Point Cloud
• OctNetFusion: Learning Depth Fusion from Data
• Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures
• Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
• Interactive 3D Modeling with a Generative Adversarial Network
• Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes
• Remote Sensing Novel View Synthesis with Implicit Multiplane Representations
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• Cascaded Refinement Network for Point Cloud Completion
• Geometric Prior Based Deep Human Point Cloud Geometry Compression
• Flex-Convolution (Million-Scale Point-Cloud Learning Beyond Grid-Worlds)
• Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
• 3D Shape Synthesis for Conceptual Design and Optimization Using Variational Autoencoders
• Learning Shape Priors for Single-View 3D Completion and Reconstruction
• Inferring Point Clouds from Single Monocular Images by Depth Intermediation
• Neural Radiance Flow for 4D View Synthesis and Video Processing
• DiffGCN: Graph Convolutional Networks via Differential Operators and Algebraic Multigrid Pooling
• 3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations
• Grid-GCN for Fast and Scalable Point Cloud Learning
• A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images
• Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
• MeshWalker: Deep Mesh Understanding by Random Walks
• CvxNet: Learnable Convex Decomposition
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds
• Learning 3D Shapes as Multi-Layered Height-maps using 2D Convolutional Networks
• Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations
• Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds
• Geometric features for voxel-based surface recognition
• Local Deep Implicit Functions for 3D Shape
• LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks
• PVRNet: Point-View Relation Neural Network for 3D Shape Recognition
• Discrete Rotation Equivariance for Point Cloud Recognition
• IPC-Net: 3D point-cloud segmentation using deep inter-point convolutional layers
• 3D Object Classification via Spherical Projections
• LapSeg3D: Weakly Supervised Semantic Segmentation of Point Clouds Representing Laparoscopic Scenes
• What can I do here? Leveraging Deep 3D saliency and geometry for fast and scalable multiple affordance detection
• Volumetric Convolution: Automatic Representation Learning in Unit Ball
• EnzyNet: enzyme classification using 3D convolutional neural networks on spatial representation
• 3DContextNet: K-d Tree Guided Hierarchical Learning of Point Clouds Using Local and Global Contextual Cues
• Learning Permutation Invariant Representations using Memory Networks
• A Variational Feature Encoding Method of 3D Object for Probabilistic Semantic SLAM
• DECOR-GAN: 3D Shape Detailization by Conditional Refinement
• Joint Shape Representation and Classification for Detecting PDAC
• Holistic 3D Scene Understanding from a Single Image with Implicit Representation
• HRGE-Net: Hierarchical Relational Graph Embedding Network for Multi-view 3D Shape Recognition
• Multi-Fingered Grasp Planning via Inference in Deep Neural Networks
• Defo-Net: Learning Body Deformation using Generative Adversarial Networks
• MG-SAGC: A multiscale graph and its self-adaptive graph convolution network for 3D point clouds
• Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds
• Deep Autoencoder for Combined Human Pose Estimation and body Model Upscaling
• VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification
• A Deeper Look at 3D Shape Classifiers
• MeshCNN Fundamentals: Geometric Learning through a Reconstructable Representation
• A Variational Observation Model of 3D Object for Probabilistic Semantic SLAM
• libmolgrid: GPU Accelerated Molecular Gridding for Deep Learning Applications
• Hierarchical View Predictor: Unsupervised 3D Global Feature Learning through Hierarchical Prediction among Unordered Views
• Cascaded Refinement Network for Point Cloud Completion with Self-supervision
• DeepCompress: Efficient Point Cloud Geometry Compression
• RocNet: Recursive Octree Network for Efficient 3D Deep Representation