Lipschitz Generative Adversarial Nets

Abstract

In this paper, we study the convergence of generative adversarial networks (GANs) from the perspective of the informativeness of the gradient of the optimal discriminative function. We show that GANs without restriction on the discriminative function space commonly suffer from the problem that the gradient produced by the discriminator is uninformative to guide the generator. By contrast, Wasserstein GAN (WGAN), where the discriminative function is restricted to 1-Lipschitz, does not suffer from such a gradient uninformativeness problem. We further show in the paper that the model with a compact dual form of Wasserstein distance, where the Lipschitz condition is relaxed, may also theoretically suffer from this issue. This implies the importance of Lipschitz condition and motivates us to study the general formulation of GANs with Lipschitz constraint, which leads to a new family of GANs that we call Lipschitz GANs (LGANs). We show that LGANs guarantee the existence and uniqueness of the optimal discriminative function as well as the existence of a unique Nash equilibrium. We prove that LGANs are generally capable of eliminating the gradient uninformativeness problem. According to our empirical analysis, LGANs are more stable and generate consistently higher quality samples compared with WGAN.

Published as a conference paper at ICML 2019

References in corpus (8)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• On Convergence and Stability of GANs
• The Numerics of GANs
• On the regularization of Wasserstein GANs
• Is Generator Conditioning Causally Related to GAN Performance?

Cited by in corpus (27)

• ViTGAN: Training GANs with Vision Transformers
• A Systematic Survey of Regularization and Normalization in GANs
• Quaternion Generative Adversarial Networks
• GANs May Have No Nash Equilibria
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Adversarial Lipschitz Regularization
• Some Theoretical Insights into Wasserstein GANs
• A case for new neural network smoothness constraints
• Adversarial symmetric GANs: bridging adversarial samples and adversarial networks
• Improving the Speed and Quality of GAN by Adversarial Training
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Group Equivariant Generative Adversarial Networks
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Self-Supervised GANs with Label Augmentation
• Designing GANs: A Likelihood Ratio Approach
• Markov-Lipschitz Deep Learning
• Approximating Lipschitz continuous functions with GroupSort neural networks
• PriorGAN: Real Data Prior for Generative Adversarial Nets
• Deep Manifold Transformation for Nonlinear Dimensionality Reduction
• Optimal 1-Wasserstein Distance for WGANs
• Distributional Robustness with IPMs and links to Regularization and GANs
• Variation Network: Learning High-level Attributes for Controlled Input Manipulation
• Moreau-Yosida $f$-divergences
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation