Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

Abstract

We address the problem of learning accurate 3D shape and camera pose from a collection of unlabeled category-specific images. We train a convolutional network to predict both the shape and the pose from a single image by minimizing the reprojection error: given several views of an object, the projections of the predicted shapes to the predicted camera poses should match the provided views. To deal with pose ambiguity, we introduce an ensemble of pose predictors which we then distill to a single "student" model. To allow for efficient learning of high-fidelity shapes, we represent the shapes by point clouds and devise a formulation allowing for differentiable projection of these. Our experiments show that the distilled ensemble of pose predictors learns to estimate the pose accurately, while the point cloud representation allows to predict detailed shape models. The supplementary video can be found at https://www.youtube.com/watch?v=LuIGovKeo60

References in corpus (4)

• TensorFlow: A system for large-scale machine learning
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Unsupervised Learning of 3D Structure from Images
• Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Cited by in corpus (7)

• A survey of synthetic data augmentation methods in computer vision
• 3D Shape Reconstruction from Vision and Touch
• Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
• Deep unsupervised 3D human body reconstruction from a sparse set of landmarks
• Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images using a View-based Representation
• Active 3D Shape Reconstruction from Vision and Touch
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics