Approximability of Discriminators Implies Diversity in GANs

Abstract

While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse. The theoretical work of Arora et al. suggests a dilemma about GANs' statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse. By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators). For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence. This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes. Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

References in corpus (10)

• Empirical Evaluation of Rectified Activations in Convolutional Network
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Adversarially Learned Inference
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Optimizing the Latent Space of Generative Networks
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• On the Discrimination-Generalization Tradeoff in GANs
• Understanding GANs: the LQG Setting
• Generative Adversarial Parallelization

Cited by in corpus (22)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• A Selective Overview of Deep Learning
• How Well Generative Adversarial Networks Learn Distributions
• Robust Estimation and Generative Adversarial Nets
• 2-Wasserstein Approximation via Restricted Convex Potentials with Application to Improved Training for GANs
• An error analysis of generative adversarial networks for learning distributions
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• The Inductive Bias of Restricted f-GANs
• Generalization Properties of Optimal Transport GANs with Latent Distribution Learning
• Classification and reconstruction of optical quantum states with deep neural networks
• SGD Learns One-Layer Networks in WGANs
• Globally Injective ReLU Networks
• Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks
• Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective
• Generalization and Memorization: The Bias Potential Model
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• Deconstructing Generative Adversarial Networks
• Making Method of Moments Great Again? -- How can GANs learn distributions
• On the Anomalous Generalization of GANs
• Online Kernel based Generative Adversarial Networks
• Non-Asymptotic Error Bounds for Bidirectional GANs