Semantic Bottleneck Scene Generation

Abstract

Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them to learn the scene structure. During inference, our model first synthesizes a realistic segmentation layout from scratch, then synthesizes a realistic scene conditioned on that layout. For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts. For the latter, we use a conditional segmentation-to-image synthesis network that captures the distribution of photo-realistic images conditioned on the semantic layout. When trained end-to-end, the resulting model outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and user-study evaluations. Moreover, we demonstrate the generated segmentation maps can be used as additional training data to strongly improve recent segmentation-to-image synthesis networks.

References in corpus (12)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Categorical Reparameterization with Gumbel-Softmax
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Learning What and Where to Draw
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• NetGAN: Generating Graphs via Random Walks
• Assessing Generative Models via Precision and Recall
• Adversarial Ranking for Language Generation
• Neural Text Generation: Past, Present and Beyond

Cited by in corpus (5)

• Generative Adversarial Transformers
• Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
• Compositional Transformers for Scene Generation
• Low Distortion Block-Resampling with Spatially Stochastic Networks
• MeronymNet: A Hierarchical Approach for Unified and Controllable Multi-Category Object Generation