Transformation-Grounded Image Generation Network for Novel 3D View Synthesis

Abstract

We present a transformation-grounded image generation network for novel 3D view synthesis from a single image. Instead of taking a 'blank slate' approach, we first explicitly infer the parts of the geometry visible both in the input and novel views and then re-cast the remaining synthesis problem as image completion. Specifically, we both predict a flow to move the pixels from the input to the novel view along with a novel visibility map that helps deal with occulsion/disocculsion. Next, conditioned on those intermediate results, we hallucinate (infer) parts of the object invisible in the input image. In addition to the new network structure, training with a combination of adversarial and perceptual loss results in a reduction in common artifacts of novel view synthesis such as distortions and holes, while successfully generating high frequency details and preserving visual aspects of the input image. We evaluate our approach on a wide range of synthetic and real examples. Both qualitative and quantitative results show our method achieves significantly better results compared to existing methods.

arXiv commentTo appear in CVPR 2017

References in corpus (24)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Improved Techniques for Training GANs
• Generating Videos with Scene Dynamics
• Deep Convolutional Inverse Graphics Network
• Autoencoding beyond pixels using a learned similarity metric
• Learning Deconvolution Network for Semantic Segmentation
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Stacked Hourglass Networks for Human Pose Estimation
• Deep multi-scale video prediction beyond mean square error
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Unsupervised Learning for Physical Interaction through Video Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Attribute2Image: Conditional Image Generation from Visual Attributes
• DeepStereo: Learning to Predict New Views from the World's Imagery
• Discriminative Regularization for Generative Models
• Generating Images Part by Part with Composite Generative Adversarial Networks
• Texture Synthesis Using Shallow Convolutional Networks with Random Filters
• Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles
• Novel Views of Objects from a Single Image
• Shape Completion Enabled Robotic Grasping

Cited by in corpus (17)

• Generative Image Inpainting with Contextual Attention
• Deferred Neural Rendering: Image Synthesis using Neural Textures
• Towards Large-Pose Face Frontalization in the Wild
• Multi-View Image Generation from a Single-View
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• Morphing and Sampling Network for Dense Point Cloud Completion
• 3D Photography using Context-aware Layered Depth Inpainting
• Geometry-aware Deep Network for Single-Image Novel View Synthesis
• DwNet: Dense warp-based network for pose-guided human video generation
• GestureGAN for Hand Gesture-to-Gesture Translation in the Wild
• Visuomotor Understanding for Representation Learning of Driving Scenes
• View Extrapolation of Human Body from a Single Image
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Learning to Caricature via Semantic Shape Transform
• Incremental Scene Synthesis
• Depth Assisted Full Resolution Network for Single Image-based View Synthesis
• Cross-View Exocentric to Egocentric Video Synthesis