Learning Representations and Generative Models for 3D Point Clouds

Abstract

Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep AutoEncoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation, as well as shape completion. We perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs, and Gaussian Mixture Models (GMMs). To quantitatively evaluate generative models we introduce measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs yield the best results overall.

Cited by in corpus (156)

• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Object-Centric Learning with Slot Attention
• A Review of Location Encoding for GeoAI: Methods and Applications
• AdvPC: Transferable Adversarial Perturbations on 3D Point Clouds
• UniGrasp: Learning a Unified Model to Grasp with Multifingered Robotic Hands
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• Unpaired Point Cloud Completion on Real Scans using Adversarial Training
• PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing
• Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules
• Neural Marching Cubes
• Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion
• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows
• Orienting Point Clouds with Dipole Propagation
• Deep Set Prediction Networks
• Learning Implicit Fields for Generative Shape Modeling
• Deep AutoEncoder-based Lossy Geometry Compression for Point Clouds
• StructureNet: Hierarchical Graph Networks for 3D Shape Generation
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• Unsupervised Generative 3D Shape Learning from Natural Images
• Learning Deformable Tetrahedral Meshes for 3D Reconstruction
• METASET: Exploring Shape and Property Spaces for Data-Driven Metamaterials Design
• Self-Supervised Few-Shot Learning on Point Clouds
• A Topology Layer for Machine Learning
• Generative 3D Part Assembly via Dynamic Graph Learning
• Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• PF-Net: Point Fractal Network for 3D Point Cloud Completion
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
• 3D Point Cloud Processing and Learning for Autonomous Driving
• IF-Defense: 3D Adversarial Point Cloud Defense via Implicit Function based Restoration
• Self-Supervised Learning for Domain Adaptation on Point-Clouds
• GRNet: Gridding Residual Network for Dense Point Cloud Completion
• Convolutional Generation of Textured 3D Meshes
• SDM-NET: Deep Generative Network for Structured Deformable Mesh
• Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks
• Morphing and Sampling Network for Dense Point Cloud Completion
• Data-Driven Approach to Encoding and Decoding 3-D Crystal Structures
• Diffusion Probabilistic Models for 3D Point Cloud Generation
• ShapeMOD: Macro Operation Discovery for 3D Shape Programs
• Conditional Set Generation with Transformers
• SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation
• Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes
• Cascaded Refinement Network for Point Cloud Completion
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• Hierarchical Poset Decoding for Compositional Generalization in Language
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• PointSCNet: Point Cloud Structure and Correlation Learning Based on Space Filling Curve-Guided Sampling
• PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors
• Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
• Unsupervised Point Cloud Pre-Training via Occlusion Completion
• Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks
• NeuralSampler: Euclidean Point Cloud Auto-Encoder and Sampler
• 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Learning Occupancy Function from Point Clouds for Surface Reconstruction
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• Higher-Order Function Networks for Learning Composable 3D Object Representations
• PointWise: An Unsupervised Point-wise Feature Learning Network
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• BSP-Net: Generating Compact Meshes via Binary Space Partitioning
• Point Cloud Completion by Skip-attention Network with Hierarchical Folding
• MOPS-Net: A Matrix Optimization-driven Network forTask-Oriented 3D Point Cloud Downsampling
• PU-GAN: a Point Cloud Upsampling Adversarial Network
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• TM-NET: Deep Generative Networks for Textured Meshes
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning
• Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction
• Conditional Single-view Shape Generation for Multi-view Stereo Reconstruction
• Learning Gradient Fields for Shape Generation
• PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
• View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions
• ChartPointFlow for Topology-Aware 3D Point Cloud Generation
• Unsupervised 3D Shape Completion through GAN Inversion
• Progressive Point Cloud Deconvolution Generation Network
• TearingNet: Point Cloud Autoencoder to Learn Topology-Friendly Representations
• RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction
• Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild
• CvxNet: Learnable Convex Decomposition
• Augmenting Implicit Neural Shape Representations with Explicit Deformation Fields
• Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds
• Combinatorial 3D Shape Generation via Sequential Assembly
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Refinement of Predicted Missing Parts Enhance Point Cloud Completion
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Particle Cloud Generation with Message Passing Generative Adversarial Networks
• COALESCE: Component Assembly by Learning to Synthesize Connections
• Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds
• Roof-GAN: Learning to Generate Roof Geometry and Relations for Residential Houses
• Multimodal Shape Completion via Conditional Generative Adversarial Networks
• Style-based Point Generator with Adversarial Rendering for Point Cloud Completion
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence
• Set-to-Sequence Methods in Machine Learning: a Review
• SoftPoolNet: Shape Descriptor for Point Cloud Completion and Classification
• Point-set Distances for Learning Representations of 3D Point Clouds
• Latent feature disentanglement for 3D meshes
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds
• GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-wise Transformations
• DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces
• Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondence
• Data augmentation and image understanding
• Deep Magnification-Flexible Upsampling over 3D Point Clouds
• CAD-PU: A Curvature-Adaptive Deep Learning Solution for Point Set Upsampling
• F-Siamese Tracker: A Frustum-based Double Siamese Network for 3D Single Object Tracking
• Temporal LiDAR Frame Prediction for Autonomous Driving
• GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
• Deep Optimized Priors for 3D Shape Modeling and Reconstruction
• DECOR-GAN: 3D Shape Detailization by Conditional Refinement
• HyperCube: Implicit Field Representations of Voxelized 3D Models
• A Point Cloud Generative Model via Tree-Structured Graph Convolutions for 3D Brain Shape Reconstruction
• Staying in Shape: Learning Invariant Shape Representations using Contrastive Learning
• 3D Brain Reconstruction by Hierarchical Shape-Perception Network from a Single Incomplete Image
• Learning Embedding of 3D models with Quadric Loss
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
• EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation
• Holistic 3D Scene Understanding from a Single Image with Implicit Representation
• Generation For Adaption: A GAN-Based Approach for 3D Domain Adaption with Point Cloud Data
• GPR-based Model Reconstruction System for Underground Utilities Using GPRNet
• 3DSNet: Unsupervised Shape-to-Shape 3D Style Transfer
• How Powerful Are Randomly Initialized Pointcloud Set Functions?
• STD-Net: Structure-preserving and Topology-adaptive Deformation Network for 3D Reconstruction from a Single Image
• Developing Creative AI to Generate Sculptural Objects
• Latent Space Representation for Shape Analysis and Learning
• Keypoint-Based Bimanual Shaping of Deformable Linear Objects under Environmental Constraints using Hierarchical Action Planning
• A Deep Signed Directional Distance Function for Object Shape Representation
• Learning Scene Dynamics from Point Cloud Sequences
• Learning to Generate 3D Shapes with Generative Cellular Automata
• Generating Unrestricted 3D Adversarial Point Clouds
• DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates
• LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
• Wireless 3D Point Cloud Delivery Using Deep Graph Neural Networks
• Discrete Point Flow Networks for Efficient Point Cloud Generation
• Unsupervised Feedforward Feature (UFF) Learning for Point Cloud Classification and Segmentation
• Machinic Surrogates: Human-Machine Relationships in Computational Creativity
• A curvature and density-based generative representation of shapes
• Point Discriminative Learning for Data-efficient 3D Point Cloud Analysis
• Branchy-GNN: a Device-Edge Co-Inference Framework for Efficient Point Cloud Processing
• Modeling 3D Surface Manifolds with a Locally Conditioned Atlas
• Top-N: Equivariant set and graph generation without exchangeability
• Deep Hybrid Self-Prior for Full 3D Mesh Generation
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction
• Neural Implicit 3D Shapes from Single Images with Spatial Patterns
• Quantifying point cloud realism through adversarially learned latent representations
• Intuitive Shape Editing in Latent Space
• 3D Shape Generation with Grid-based Implicit Functions
• Hierarchical View Predictor: Unsupervised 3D Global Feature Learning through Hierarchical Prediction among Unordered Views
• LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulation
• The Power of Points for Modeling Humans in Clothing
• NeuralQAAD: An Efficient Differentiable Framework for High Resolution Point Cloud Compression
• Learning to Represent and Predict Sets with Deep Neural Networks
• Training Deep Normalizing Flow Models in Highly Incomplete Data Scenarios with Prior Regularization
• Point Cloud Colorization Based on Densely Annotated 3D Shape Dataset
• GAMesh: Guided and Augmented Meshing for Deep Point Networks