Diversity-Sensitive Conditional Generative Adversarial Networks

Abstract

We propose a simple yet highly effective method that addresses the mode-collapse problem in the Conditional Generative Adversarial Network (cGAN). Although conditional distributions are multi-modal (i.e., having many modes) in practice, most cGAN approaches tend to learn an overly simplified distribution where an input is always mapped to a single output regardless of variations in latent code. To address such issue, we propose to explicitly regularize the generator to produce diverse outputs depending on latent codes. The proposed regularization is simple, general, and can be easily integrated into most conditional GAN objectives. Additionally, explicit regularization on generator allows our method to control a balance between visual quality and diversity. We demonstrate the effectiveness of our method on three conditional generation tasks: image-to-image translation, image inpainting, and future video prediction. We show that simple addition of our regularization to existing models leads to surprisingly diverse generations, substantially outperforming the previous approaches for multi-modal conditional generation specifically designed in each individual task.

Accepted as a conference paper at ICLR 2019

Cited by in corpus (44)

• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• A Systematic Survey of Regularization and Normalization in GANs
• Temporal Prediction and Evaluation of Brassica Growth in the Field using Conditional Generative Adversarial Networks
• Generative data-driven approaches for stochastic subgrid parameterizations in an idealized ocean model
• The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• High Perceptual Quality Image Denoising with a Posterior Sampling CGAN
• On the Evaluation of Conditional GANs
• CCVS: Context-aware Controllable Video Synthesis
• Improved Consistency Regularization for GANs
• Fast and Scalable Earth Texture Synthesis using Spatially Assembled Generative Adversarial Neural Networks
• Few-shot Image Generation via Cross-domain Correspondence
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
• SSD-GAN: Measuring the Realness in the Spatial and Spectral Domains
• Regularizing Generative Adversarial Networks under Limited Data
• Instrument-To-Instrument translation: Instrumental advances drive restoration of solar observation series via deep learning
• MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
• Diverse Semantic Image Synthesis via Probability Distribution Modeling
• DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling
• Max-Affine Spline Insights into Deep Generative Networks
• Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data
• Semantically Multi-modal Image Synthesis
• Image-to-Image Translation with Low Resolution Conditioning
• Training Generative Adversarial Networks in One Stage
• Stochastic Conditional Generative Networks with Basis Decomposition
• Inclusive GAN: Improving Data and Minority Coverage in Generative Models
• Rethinking conditional GAN training: An approach using geometrically structured latent manifolds
• SLGAN: Style- and Latent-guided Generative Adversarial Network for Desirable Makeup Transfer and Removal
• Conditional Coupled Generative Adversarial Networks for Zero-Shot Domain Adaptation
• A Generative Model for Hallucinating Diverse Versions of Super Resolution Images
• Blur, Noise, and Compression Robust Generative Adversarial Networks
• A cost-effective method for improving and re-purposing large, pre-trained GANs by fine-tuning their class-embeddings
• Adversarial sampling of unknown and high-dimensional conditional distributions
• CoPE: Conditional image generation using Polynomial Expansions
• Improving Text to Image Generation using Mode-seeking Function
• Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation
• Point-to-Point Video Generation
• Toward Zero-Shot Unsupervised Image-to-Image Translation
• Deep Sequence Learning for Video Anticipation: From Discrete and Deterministic to Continuous and Stochastic
• Explorable Tone Mapping Operators
• Learning to Generate Novel Scene Compositions from Single Images and Videos
• Noise Robust Generative Adversarial Networks