You Only Need Adversarial Supervision for Semantic Image Synthesis

Abstract

Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity with better alignment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of $6$ FID and $5$ mIoU points over the state of the art across different datasets using only adversarial supervision.

Published at ICLR 2021 (Main Conference). Code repository: https://github.com/boschresearch/OASIS

References in corpus (10)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Assessing Generative Models via Precision and Recall
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects
• Implicit Maximum Likelihood Estimation
• Rethinking Spatially-Adaptive Normalization

Cited by in corpus (12)

• C-VTON: Context-Driven Image-Based Virtual Try-On Network
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
• Compositional Transformers for Scene Generation
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Variation-Aware Semantic Image Synthesis
• Towards Photorealistic Colorization by Imagination
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• Image Synthesis via Semantic Composition
• Learning to Generate Novel Scene Compositions from Single Images and Videos
• deepNIR: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques
• Collaging Class-specific GANs for Semantic Image Synthesis