View Extrapolation of Human Body from a Single Image
arXiv:1804.04213
Abstract
We study how to synthesize novel views of human body from a single image. Though recent deep learning based methods work well for rigid objects, they often fail on objects with large articulation, like human bodies. The core step of existing methods is to fit a map from the observable views to novel views by CNNs; however, the rich articulation modes of human body make it rather challenging for CNNs to memorize and interpolate the data well. To address the problem, we propose a novel deep learning based pipeline that explicitly estimates and leverages the geometry of the underlying human body. Our new pipeline is a composition of a shape estimation network and an image generation network, and at the interface a perspective transformation is applied to generate a forward flow for pixel value transportation. Our design is able to factor out the space of data variation and makes learning at each step much easier. Empirically, we show that the performance for pose-varying objects can be improved dramatically. Our method can also be applied on real data captured by 3D sensors, and the flow generated by our methods can be used for generating high quality results in higher resolution.
arXiv commentAccepted to CVPR 2018
References in corpus (8)
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
- Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
- Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
- Multi-View Image Generation from a Single-View
- Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
- 3D-Assisted Image Feature Synthesis for Novel Views of an Object