Towards Realistic 3D Embedding via View Alignment

Abstract

Recent advances in generative adversarial networks (GANs) have achieved great success in automated image composition that generates new images by embedding interested foreground objects into background images automatically. On the other hand, most existing works deal with foreground objects in two-dimensional (2D) images though foreground objects in three-dimensional (3D) models are more flexible with 360-degree view freedom. This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable. The differential discriminator guides to learn geometric transformation from background images so that the composed 3D models can be aligned with the background images with realistic poses and views. The texture generator adopts a novel view encoding mechanism for generating accurate object textures for the 3D models under the estimated views. Extensive experiments over two synthesis tasks (car synthesis with KITTI and pedestrian synthesis with Cityscapes) show that VA-GAN achieves high-fidelity composition qualitatively and quantitatively as compared with state-of-the-art generation methods.

arXiv commentThis work has been merged into another project

References in corpus (10)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Least Squares Generative Adversarial Networks
• Few-Shot Unsupervised Image-to-Image Translation
• Context-Aware Synthesis and Placement of Object Instances
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• 3D-Aware Scene Manipulation via Inverse Graphics
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• Learning to Generate Synthetic Data via Compositing

Cited by in corpus (9)

• Blind Image Super-Resolution via Contrastive Representation Learning
• EMLight: Lighting Estimation via Spherical Distribution Approximation
• Adversarial Image Composition with Auxiliary Illumination
• FBC-GAN: Diverse and Flexible Image Synthesis via Foreground-Background Composition
• Bi-level Feature Alignment for Versatile Image Translation and Manipulation
• Spatial-Aware GAN for Unsupervised Person Re-identification
• Hierarchy Composition GAN for High-fidelity Image Synthesis
• A Survey on Adversarial Image Synthesis
• Scene Text recognition with Full Normalization