SEAN: Image Synthesis with Semantic Region-Adaptive Normalization

Abstract

We propose semantic region-adaptive normalization (SEAN), a simple but effective building block for Generative Adversarial Networks conditioned on segmentation masks that describe the semantic regions in the desired output image. Using SEAN normalization, we can build a network architecture that can control the style of each semantic region individually, e.g., we can specify one style reference image per region. SEAN is better suited to encode, transfer, and synthesize style than the best previous method in terms of reconstruction quality, variability, and visual quality. We evaluate SEAN on multiple datasets and report better quantitative metrics (e.g. FID, PSNR) than the current state of the art. SEAN also pushes the frontier of interactive image editing. We can interactively edit images by changing segmentation masks or the style for any given region. We can also interpolate styles from two reference images per region.

Accepted as a CVPR 2020 oral paper. The interactive demo is available at https://youtu.be/0Vbj9xFgoUw

References in corpus (1)

• TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures

Cited by in corpus (15)

• Barbershop: GAN-based Image Compositing using Segmentation Masks
• SofGAN: A Portrait Image Generator with Dynamic Styling
• Can segmentation models be trained with fully synthetically generated data?
• Layout-to-Image Translation with Double Pooling Generative Adversarial Networks
• Location-aware Adaptive Normalization: A Deep Learning Approach For Wildfire Danger Forecasting
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Example Forgetting: A Novel Approach to Explain and Interpret Deep Neural Networks in Seismic Interpretation
• Explaining Deep Models through Forgettable Learning Dynamics
• Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging
• FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing
• EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models
• Semantic Image Synthesis via Class-Adaptive Cross-Attention
• Controllable Face Synthesis with Semantic Latent Diffusion Models
• LatentKeypointGAN: Controlling Images via Latent Keypoints
• SAT3D: Image-driven Semantic Attribute Transfer in 3D