Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Abstract

We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.

References in corpus (2)

• Geometric deep learning: going beyond Euclidean data
• SurfNet: Generating 3D shape surfaces using deep residual networks

Cited by in corpus (42)

• Contrastive Multi-View Representation Learning on Graphs
• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Adversarial Attack and Defense on Point Sets
• pixelNeRF: Neural Radiance Fields from One or Few Images
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
• 3D Hand Shape and Pose Estimation from a Single RGB Image
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Adversarial Generation of Continuous Implicit Shape Representations
• 3D Shape Reconstruction from Vision and Touch
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attention
• DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
• Coherent Point Drift Networks: Unsupervised Learning of Non-Rigid Point Set Registration
• RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis
• DOVE: Learning Deformable 3D Objects by Watching Videos
• Neural Radiance Flow for 4D View Synthesis and Video Processing
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images
• Convolutional Neural Networks on non-uniform geometrical signals using Euclidean spectral transformation
• Differentiable Rendering with Perturbed Optimizers
• Neural Star Domain as Primitive Representation
• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
• A Simple and Scalable Shape Representation for 3D Reconstruction
• Local Deep Implicit Functions for 3D Shape
• Unsupervised object-centric video generation and decomposition in 3D
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
• Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
• Learning View Priors for Single-view 3D Reconstruction
• Scan2Mesh: From Unstructured Range Scans to 3D Meshes
• Active 3D Shape Reconstruction from Vision and Touch
• Learning Embedding of 3D models with Quadric Loss
• Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction
• D-OccNet: Detailed 3D Reconstruction Using Cross-Domain Learning
• Modeling Varying Camera-IMU Time Offset in Optimization-Based Visual-Inertial Odometry
• Geodesic-HOF: 3D Reconstruction Without Cutting Corners
• A Novel Self-Intersection Penalty Term for Statistical Body Shape Models and Its Applications in 3D Pose Estimation
• Deep Hybrid Self-Prior for Full 3D Mesh Generation
• Domain Adaptation for Real-World Single View 3D Reconstruction
• Single Image 3D Object Estimation with Primitive Graph Networks
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction
• Synthesizing Photorealistic Images with Deep Generative Learning