Smoothness and Stability in GANs

Abstract

Generative adversarial networks, or GANs, commonly display unstable behavior during training. In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs. In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisfied by the divergence that is minimized by the GAN and the generator's architecture. We find that existing GAN variants satisfy some, but not all, of these conditions. Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulfills these conditions simultaneously. In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.

ICLR 2020

References in corpus (10)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Which Training Methods for GANs do actually Converge?
• Generative Moment Matching Networks
• On Convergence and Stability of GANs
• On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
• On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradient
• On the Limitations of First-Order Approximation in GAN Dynamics
• Probability Functional Descent: A Unifying Perspective on GANs, Variational Inference, and Reinforcement Learning

Cited by in corpus (8)

• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Group Equivariant Generative Adversarial Networks
• Generalized Energy Based Models
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Frank-Wolfe Methods in Probability Space
• Conditional Generative Modeling via Learning the Latent Space
• Data Augmentation via Mixed Class Interpolation using Cycle-Consistent Generative Adversarial Networks Applied to Cross-Domain Imagery
• The equivalence between Stein variational gradient descent and black-box variational inference