Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis

Abstract

Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we propose to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps and eventually generate the images. Moreover, we propose a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts than previous multi-scale discriminators. We achieve state-of-the-art results on both quantitative metrics and subjective evaluation on various semantic segmentation datasets, demonstrating the effectiveness of our approach.

Accepted by NeurIPS 2019. Code is available soon at https://github.com/xh-liu/CC-FPSE

Cited by in corpus (33)

• A Survey of Human-in-the-loop for Machine Learning
• You Only Need Adversarial Supervision for Semantic Image Synthesis
• SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects
• Dynamic Neural Networks: A Survey
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data
• Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images
• Responsible Disclosure of Generative Models Using Scalable Fingerprinting
• Compositional Transformers for Scene Generation
• FaR-GAN for One-Shot Face Reenactment
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation
• COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• Few-shot Semantic Image Synthesis Using StyleGAN Prior
• TSIT: A Simple and Versatile Framework for Image-to-Image Translation
• Guided Deep Decoder: Unsupervised Image Pair Fusion
• Dual Contrastive Loss and Attention for GANs
• Diverse Semantic Image Synthesis via Probability Distribution Modeling
• Dual Attention GANs for Semantic Image Synthesis
• Learning Semantic Person Image Generation by Region-Adaptive Normalization
• Diversifying Semantic Image Synthesis and Editing via Class- and Layer-wise VAEs
• World-Consistent Video-to-Video Synthesis
• Text-to-Image Generation Grounded by Fine-Grained User Attention
• Low Distortion Block-Resampling with Spatially Stochastic Networks
• An Attention Module for Convolutional Neural Networks
• Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis
• Semantic Editing On Segmentation Map Via Multi-Expansion Loss
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives
• Human-In-The-Loop Document Layout Analysis
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• Image Synthesis via Semantic Composition
• Collaging Class-specific GANs for Semantic Image Synthesis
• Stay Positive: Non-Negative Image Synthesis for Augmented Reality