Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images

Abstract

Recovering the 3D shape of an object from single or multiple images with deep neural networks has been attracting increasing attention in the past few years. Mainstream works (e.g. 3D-R2N2) use recurrent neural networks (RNNs) to sequentially fuse feature maps of input images. However, RNN-based approaches are unable to produce consistent reconstruction results when given the same input images with different orders. Moreover, RNNs may forget important features from early input images due to long-term memory loss. To address these issues, we propose a novel framework for single-view and multi-view 3D object reconstruction, named Pix2Vox++. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. A multi-scale context-aware fusion module is then introduced to adaptively select high-quality reconstructions for different parts from all coarse 3D volumes to obtain a fused 3D volume. To further correct the wrongly recovered parts in the fused 3D volume, a refiner is adopted to generate the final output. Experimental results on the ShapeNet, Pix3D, and Things3D benchmarks show that Pix2Vox++ performs favorably against state-of-the-art methods in terms of both accuracy and efficiency.

International Journal of Computer Vision (IJCV). arXiv admin note: text overlap with arXiv:1901.11153

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the difficulty of training Recurrent Neural Networks
• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era
• Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images
• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction
• Dense 3D Object Reconstruction from a Single Depth View
• Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers
• StructureNet: Hierarchical Graph Networks for 3D Shape Generation
• Single stream parallelization of generalized LSTM-like RNNs on a GPU
• SCORES: Shape Composition with Recursive Substructure Priors

Cited by in corpus (13)

• CSDN: Cross-modal Shape-transfer Dual-refinement Network for Point Cloud Completion
• SoftPool++: An Encoder-Decoder Network for Point Cloud Completion
• GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff
• HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction
• Dense Voxel 3D Reconstruction Using a Monocular Event Camera
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• Single-View 3D Object Reconstruction from Shape Priors in Memory
• Mesh deformation-based single-view 3D reconstruction of thin eyeglasses frames with differentiable rendering
• Glissando-Net: Deep sinGLe vIew category level poSe eStimation ANd 3D recOnstruction
• A Conditional Point Cloud Diffusion Model for Deformable Liver Motion Tracking Via a Single Arbitrarily-Angled X-ray Projection
• Shape2.5D: A Dataset of Texture-less Surfaces for Depth and Normals Estimation
• LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction
• Component Selection for Craft Assembly Tasks