Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Abstract

3D reconstruction is a longstanding ill-posed problem, which has been explored for decades by the computer vision, computer graphics, and machine learning communities. Since 2015, image-based 3D reconstruction using convolutional neural networks (CNN) has attracted increasing interest and demonstrated an impressive performance. Given this new era of rapid evolution, this article provides a comprehensive survey of the recent developments in this field. We focus on the works which use deep learning techniques to estimate the 3D shape of generic objects either from a single or multiple RGB images. We organize the literature based on the shape representations, the network architectures, and the training mechanisms they use. While this survey is intended for methods which reconstruct generic objects, we also review some of the recent works which focus on specific object classes such as human body shapes and faces. We provide an analysis and comparison of the performance of some key papers, summarize some of the open problems in this field, and discuss promising directions for future research.

arXiv admin note: text overlap with arXiv:1806.06098, arXiv:1712.06584, arXiv:1804.10975 by other authors

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• ShapeNet: An Information-Rich 3D Model Repository
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images
• MarrNet: 3D Shape Reconstruction via 2.5D Sketches
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• Dense 3D Object Reconstruction from a Single Depth View
• 3D Sketching using Multi-View Deep Volumetric Prediction
• Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes
• Learning to Reconstruct Shapes from Unseen Classes
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• Numerical Inversion of SRNF Maps for Elastic Shape Analysis of Genus-Zero Surfaces
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
• Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer
• A Survey on Deep Learning Architectures for Image-based Depth Reconstruction
• Compact Model Representation for 3D Reconstruction
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• Single-view Object Shape Reconstruction Using Deep Shape Prior and Silhouette
• Inferring Point Clouds from Single Monocular Images by Depth Intermediation

Cited by in corpus (29)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Artificial Intelligence in the Creative Industries: A Review
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images
• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• Guided Depth Map Super-resolution: A Survey
• Damage detection using in-domain and cross-domain transfer learning
• Deep Learning for 3D Point Clouds: A Survey
• A topological solution to object segmentation and tracking
• Res-U2Net: Untrained Deep Learning for Phase Retrieval and Image Reconstruction
• DUDE: Deep Unsigned Distance Embeddings for Hi-Fidelity Representation of Complex 3D Surfaces
• Generating 3D Bio-Printable Patches Using Wound Segmentation and Reconstruction to Treat Diabetic Foot Ulcers
• Neural Network Methods for Radiation Detectors and Imaging
• Estimating and abstracting the 3D structure of bones using neural networks on X-ray (2D) images
• Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity
• Machine Learning for Detection of 3D Features using sparse X-ray data
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• An unsupervised deep learning framework for medical image denoising
• Dexterous Pre-grasp Manipulation for Human-like Functional Categorical Grasping: Deep Reinforcement Learning and Grasp Representations
• A Study into Echocardiography View Conversion
• Teat Pose Estimation via RGBD Segmentation for Automated Milking
• Mesh deformation-based single-view 3D reconstruction of thin eyeglasses frames with differentiable rendering
• PointBA: Towards Backdoor Attacks in 3D Point Cloud
• A Revisit of Shape Editing Techniques: from the Geometric to the Neural Viewpoint
• Skeleton Merger: an Unsupervised Aligned Keypoint Detector
• Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images
• An Edge Computing-based Photo Crowdsourcing Framework for Real-time 3D Reconstruction
• Holodeck: Immersive 3D Displays Using Swarms of Flying Light Specks
• In-Hand 3D Object Scanning from an RGB Sequence