First Order Generative Adversarial Networks
arXiv:1802.04591
Abstract
GANs excel at learning high dimensional distributions, but they can update generator parameters in directions that do not correspond to the steepest descent direction of the objective. Prominent examples of problematic update directions include those used in both Goodfellow's original GAN and the WGAN-GP. To formally describe an optimal update direction, we introduce a theoretical framework which allows the derivation of requirements on both the divergence and corresponding method for determining an update direction, with these requirements guaranteeing unbiased mini-batch updates in the direction of steepest descent. We propose a novel divergence which approximates the Wasserstein distance while regularizing the critic's first order information. Together with an accompanying update direction, this divergence fulfills the requirements for unbiased steepest descent updates. We verify our method, the First Order GAN, with image generation on CelebA, LSUN and CIFAR-10 and set a new state of the art on the One Billion Word language generation task. Code to reproduce experiments is available.
arXiv commentAccepted to 35th International Conference on Machine Learning (ICML). Code to reproduce experiments is available https://github.com/zalandoresearch/first_order_gan
References in corpus (14)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Improved Techniques for Training GANs
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Which Training Methods for GANs do actually Converge?
- f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
- VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
- Unrolled Generative Adversarial Networks
- Generative Moment Matching Networks
- The Cramer Distance as a Solution to Biased Wasserstein Gradients
- Demystifying MMD GANs
- On the regularization of Wasserstein GANs
- Texture Synthesis with Spatial Generative Adversarial Networks
- Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
- Gradient descent GAN optimization is locally stable