DOVE: Learning Deformable 3D Objects by Watching Videos

Abstract

Learning deformable 3D objects from 2D images is often an ill-posed problem. Existing methods rely on explicit supervision to establish multi-view correspondences, such as template shape models and keypoint annotations, which restricts their applicability on objects "in the wild". A more natural way of establishing correspondences is by watching videos of objects moving around. In this paper, we present DOVE, a method that learns textured 3D models of deformable object categories from monocular videos available online, without keypoint, viewpoint or template shape supervision. By resolving symmetry-induced pose ambiguities and leveraging temporal correspondences in videos, the model automatically learns to factor out 3D shape, articulated pose and texture from each individual RGB frame, and is ready for single-image inference at test time. In the experiments, we show that existing methods fail to learn sensible 3D shapes without additional keypoint or template supervision, whereas our method produces temporally consistent 3D models, which can be animated and rendered from arbitrary viewpoints.

arXiv commentProject Page: https://dove3d.github.io/

References in corpus (18)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Unsupervised Learning of Depth and Ego-Motion from Video
• On Learning 3D Face Morphable Model from In-the-wild Images
• Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Accelerating 3D Deep Learning with PyTorch3D
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• Sim2real transfer learning for 3D human pose estimation: motion to the rescue
• Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations
• PVA: Pixel-aligned Volumetric Avatars
• Online Adaptation for Consistent Mesh Reconstruction in the Wild
• Unsupervised 3D Pose Estimation with Geometric Self-Supervision
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
• Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
• FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling
• Neural Articulated Radiance Field