Understanding and Stabilizing GANs' Training Dynamics with Control Theory
arXiv:1909.13188
Abstract
Generative adversarial networks (GANs) are effective in generating realistic images but the training is often unstable. There are existing efforts that model the training dynamics of GANs in the parameter space but the analysis cannot directly motivate practically effective stabilizing methods. To this end, we present a conceptually novel perspective from control theory to directly model the dynamics of GANs in the function space and provide simple yet effective methods to stabilize GANs' training. We first analyze the training dynamic of a prototypical Dirac GAN and adopt the widely-used closed-loop control (CLC) to improve its stability. We then extend CLC to stabilize the training dynamic of normal GANs, where CLC is implemented as a squared regularizer on the output of the discriminator. Empirical results show that our method can effectively stabilize the training and obtain state-of-the-art performance on data generation tasks.
References in corpus (18)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Which Training Methods for GANs do actually Converge?
- Unrolled Generative Adversarial Networks
- Large Scale Adversarial Representation Learning
- Triple Generative Adversarial Nets
- LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
- The Numerics of GANs
- Dual Discriminator Generative Adversarial Nets
- Consistency Regularization for Generative Adversarial Networks
- Generative Adversarial Network Training is a Continual Learning Problem
- Understanding GANs: the LQG Setting
- Frank-Wolfe Algorithms for Saddle Point Problems
- Negative Momentum for Improved Game Dynamics
- Composite Functional Gradient Learning of Generative Adversarial Models
- Learning Implicit Generative Models by Teaching Explicit Ones