Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image

Abstract

An emerging problem in computer vision is the reconstruction of 3D shape and pose of an object from a single image. Hitherto, the problem has been addressed through the application of canonical deep learning methods to regress from the image directly to the 3D shape and pose labels. These approaches, however, are problematic from two perspectives. First, they are minimizing the error between 3D shapes and pose labels - with little thought about the nature of this label error when reprojecting the shape back onto the image. Second, they rely on the onerous and ill-posed task of hand labeling natural images with respect to 3D shape and pose. In this paper we define the new task of pose-aware shape reconstruction from a single image, and we advocate that cheaper 2D annotations of objects silhouettes in natural images can be utilized. We design architectures of pose-aware shape reconstruction which re-project the predicted shape back on to the image using the predicted pose. Our evaluation on several object categories demonstrates the superiority of our method for predicting pose-aware 3D shapes from natural images.

First sub

References in corpus (6)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Autoencoding beyond pixels using a learned similarity metric
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• 3D Shape Induction from 2D Views of Multiple Objects
• Crafting a multi-task CNN for viewpoint estimation
• 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Cited by in corpus (15)

• DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency
• MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image
• GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision
• A Simple and Scalable Shape Representation for 3D Reconstruction
• 3D Shape Reconstruction from a Single 2D Image via 2D-3D Self-Consistency
• Learning View Priors for Single-view 3D Reconstruction
• Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network
• From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks
• 3D Surface Reconstruction by Pointillism
• What Do Single-view 3D Reconstruction Networks Learn?