DOVE: Learning Deformable 3D Objects by Watching Videos

Abstract

Learning deformable 3D objects from 2D images is often an ill-posed problem. Existing methods rely on explicit supervision to establish multi-view correspondences, such as template shape models and keypoint annotations, which restricts their applicability on objects "in the wild". A more natural way of establishing correspondences is by watching videos of objects moving around. In this paper, we present DOVE, a method that learns textured 3D models of deformable object categories from monocular videos available online, without keypoint, viewpoint or template shape supervision. By resolving symmetry-induced pose ambiguities and leveraging temporal correspondences in videos, the model automatically learns to factor out 3D shape, articulated pose and texture from each individual RGB frame, and is ready for single-image inference at test time. In the experiments, we show that existing methods fail to learn sensible 3D shapes without additional keypoint or template supervision, whereas our method produces temporally consistent 3D models, which can be animated and rendered from arbitrary viewpoints.

Project Page: https://dove3d.github.io/

References in corpus (11)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Accelerating 3D Deep Learning with PyTorch3D
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• PVA: Pixel-aligned Volumetric Avatars
• Unsupervised 3D Pose Estimation with Geometric Self-Supervision
• Online Adaptation for Consistent Mesh Reconstruction in the Wild
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
• FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling
• Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
• Neural Articulated Radiance Field