Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs
arXiv:2011.00844
Abstract
Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.
Accepted to ICLR2021 as an oral presentation. Unsupervised 3D reconstruction via 2D GANs
References in corpus (6)
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Unsupervised Generative 3D Shape Learning from Natural Images
- Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
- Real or Not Real, that is the Question
- Implicit Mesh Reconstruction from Unannotated Image Collections
- Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency