Generative Image Modeling using Style and Structure Adversarial Networks
arXiv:1603.05631
Abstract
Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our S^2-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.
References in corpus (9)
- Conditional Generative Adversarial Nets
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
- Pixel Recurrent Neural Networks
- DRAW: A Recurrent Neural Network For Image Generation
- Autoencoding beyond pixels using a learned similarity metric
- Generative Moment Matching Networks
- Deep multi-scale video prediction beyond mean square error
- Generating Images from Captions with Attention