Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Abstract

Conventional methods of 3D object generative modeling learn volumetric predictions using deep networks with 3D convolutional operations, which are direct analogies to classical 2D ones. However, these methods are computationally wasteful in attempt to predict 3D shapes, where information is rich only on the surfaces. In this paper, we propose a novel 3D generative modeling framework to efficiently generate object shapes in the form of dense point clouds. We use 2D convolutional operations to predict the 3D structure from multiple viewpoints and jointly apply geometric reasoning with 2D projection optimization. We introduce the pseudo-renderer, a differentiable module to approximate the true rendering operation, to synthesize novel depth maps for optimization. Experimental results for single-image 3D object reconstruction tasks show that we outperforms state-of-the-art methods in terms of shape similarity and prediction density.

Cited by in corpus (75)

• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Differentiable Surface Splatting for Point-based Geometry Processing
• Dense 3D Object Reconstruction from a Single Depth View
• Differentiable Rendering: A Survey
• Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
• PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion
• Learning Implicit Fields for Generative Shape Modeling
• 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image
• GRNet: Gridding Residual Network for Dense Point Cloud Completion
• AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation
• ADOP: Approximate Differentiable One-Pixel Point Rendering
• Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point Clouds of Wild Scenes
• Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
• Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks
• SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images
• Geometric Capsule Autoencoders for 3D Point Clouds
• Higher-Order Function Networks for Learning Composable 3D Object Representations
• BSP-Net: Generating Compact Meshes via Binary Space Partitioning
• Convolutional Occupancy Networks
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• Conditional Single-view Shape Generation for Multi-view Stereo Reconstruction
• A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images
• Compositionally Generalizable 3D Structure Prediction
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• CvxNet: Learnable Convex Decomposition
• Semantic Correspondence via 2D-3D-2D Cycle
• Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds
• DDR-Net: Learning Multi-Stage Multi-View Stereo With Dynamic Depth Range
• MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image
• Point-set Distances for Learning Representations of 3D Point Clouds
• Object-Centric Multi-View Aggregation
• Learning Structural Graph Layouts and 3D Shapes for Long Span Bridges 3D Reconstruction
• SkeletonNet: A Topology-Preserving Solution for Learning Mesh Reconstruction of Object Surfaces from RGB Images
• LightSAL: Lightweight Sign Agnostic Learning for Implicit Surface Representation
• 3D-GMNet: Single-View 3D Shape Recovery as A Gaussian Mixture
• Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations
• Semantic Photometric Bundle Adjustment on Natural Sequences
• GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
• Unsupervised Learning of Intrinsic Structural Representation Points
• SeqXY2SeqZ: Structure Learning for 3D Shapes by Sequentially Predicting 1D Occupancy Segments From 2D Coordinates
• Learning Depth With Very Sparse Supervision
• EC-Net: an Edge-aware Point set Consolidation Network
• HyperCube: Implicit Field Representations of Voxelized 3D Models
• Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network
• Learning 3D Part Assembly from a Single Image
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images
• From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• SE-MD: A Single-encoder multiple-decoder deep network for point cloud generation from 2D images
• Holistic 3D Scene Understanding from a Single Image with Implicit Representation
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• I-nteract 2.0: A Cyber-Physical System to Design 3D Models using Mixed Reality Technologies and Deep Learning for Additive Manufacturing
• End-to-end Malaria Diagnosis and 3D Cell Rendering with Deep Learning
• DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization
• Inferring 3D Shapes from Image Collections using Adversarial Networks
• Learning geometry-image representation for 3D point cloud generation
• Deep Feedback Inverse Problem Solver
• Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry
• D-OccNet: Detailed 3D Reconstruction Using Cross-Domain Learning
• ViewSynth: Learning Local Features from Depth using View Synthesis
• Novel View Synthesis from a Single Image via Unsupervised learning
• Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset
• Neural Implicit 3D Shapes from Single Images with Spatial Patterns
• OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics
• PointManifoldCut: Point-wise Augmentation in the Manifold for Point Clouds
• Learning to Reconstruct and Segment 3D Objects
• Irregular Convolutional Auto-Encoder on Point Clouds
• GAMesh: Guided and Augmented Meshing for Deep Point Networks
• Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images
• What Do Single-view 3D Reconstruction Networks Learn?
• Domain Adaptation for Real-World Single View 3D Reconstruction
• The Power of Points for Modeling Humans in Clothing