Semantic Image Synthesis with Spatially-Adaptive Normalization

Abstract

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .

arXiv commentAccepted as a CVPR 2019 oral paper

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Which Training Methods for GANs do actually Converge?
• A Learned Representation For Artistic Style
• Modulating early visual processing by language
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• On Self Modulation for Generative Adversarial Networks
• Geometric GAN
• Learning Visual Reasoning Without Strong Priors

Cited by in corpus (50)

• Generative Adversarial Network in Medical Imaging: A Review
• Media Forensics and DeepFakes: an overview
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Stochastic Super-Resolution for Downscaling Time-Evolving Atmospheric Fields with a Generative Adversarial Network
• pixelNeRF: Neural Radiance Fields from One or Few Images
• medigan: a Python library of pretrained generative models for medical image synthesis
• WaveGrad: Estimating Gradients for Waveform Generation
• Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
• Dynamic Neural Networks: A Survey
• Bipartite Graph Reasoning GANs for Person Image Generation
• Text as Neural Operator: Image Manipulation by Text Instruction
• One-shot Face Reenactment
• A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities
• One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
• Towards Realistic 3D Embedding via View Alignment
• Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
• A Convolutional LSTM based Residual Network for Deepfake Video Detection
• Rethinking Spatially-Adaptive Normalization
• Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
• Few-shot Video-to-Video Synthesis
• Deep learning-based topological optimization for representing a user-specified design area
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• KG-GAN: Knowledge-Guided Generative Adversarial Networks
• Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation
• cGANs with Conditional Convolution Layer
• MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation
• Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs
• Adversarial Attacks on Co-Occurrence Features for GAN Detection
• Dual Attention GANs for Semantic Image Synthesis
• Boosting Image Outpainting with Semantic Layout Prediction
• Context-Aware Image Inpainting with Learned Semantic Priors
• Sentence Guided Temporal Modulation for Dynamic Video Thumbnail Generation
• Red-GAN: Attacking class imbalance via conditioned generation. Yet another perspective on medical image synthesis for skin lesion dermoscopy and brain tumor MRI
• Decomposing Normal and Abnormal Features of Medical Images for Content-based Image Retrieval
• Systematic Analysis and Removal of Circular Artifacts for StyleGAN
• Fashion Editing with Adversarial Parsing Learning
• Semantic Image Manipulation Using Scene Graphs
• Deep Consensus Learning
• Stylized Neural Painting
• Relationship-Aware Spatial Perception Fusion for Realistic Scene Layout Generation
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• MetalGAN: a Cluster-based Adaptive Training for Few-Shot Adversarial Colorization
• CanvasVAE: Learning to Generate Vector Graphic Documents
• Noise-Powered Disentangled Representation for Unsupervised Speckle Reduction of Optical Coherence Tomography Images
• Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition
• Conditional Invertible Neural Networks for Diverse Image-to-Image Translation
• Delving into Rectifiers in Style-Based Image Translation
• Synthesizing Photorealistic Images with Deep Generative Learning
• Collaging Class-specific GANs for Semantic Image Synthesis
• Bridging the Gap between Label- and Reference-based Synthesis in Multi-attribute Image-to-Image Translation