Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis

Abstract

With the remarkable recent progress on learning deep generative models, it becomes increasingly interesting to develop models for controllable image synthesis from reconfigurable inputs. This paper focuses on a recent emerged task, layout-to-image, to learn generative models that are capable of synthesizing photo-realistic images from spatial layout (i.e., object bounding boxes configured in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors). This paper first proposes an intuitive paradigm for the task, layout-to-mask-to-image, to learn to unfold object masks of given bounding boxes in an input layout to bridge the gap between the input layout and synthesized images. Then, this paper presents a method built on Generative Adversarial Networks for the proposed layout-to-mask-to-image with style control at both image and mask levels. Object masks are learned from the input layout and iteratively refined along stages in the generator network. Style control at the image level is the same as in vanilla GANs, while style control at the object mask level is realized by a proposed novel feature normalization scheme, Instance-Sensitive and Layout-Aware Normalization. In experiments, the proposed method is tested in the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained.

arXiv comment16 pages (w/o ref), 15 figures)

References in corpus (21)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Toward Multimodal Image-to-Image Translation
• A Learned Representation For Artistic Style
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Adversarially Learned Inference
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Modulating early visual processing by language
• A Theory of Generative ConvNet
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Classification Accuracy Score for Conditional Generative Models
• Geometric GAN
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph

Cited by in corpus (4)

• Adversarial Text-to-Image Synthesis: A Review
• Generating unseen complex scenes: are we there yet?
• High-Resolution Complex Scene Synthesis with Transformers
• Deep Consensus Learning