On the Limitations of First-Order Approximation in GAN Dynamics

Abstract

While Generative Adversarial Networks (GANs) have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, both in theory and in practice. To address this issue, we study GAN dynamics in a simple yet rich parametric model that exhibits several of the common problematic convergence behaviors such as vanishing gradients, mode collapse, and diverging or oscillatory behavior. In spite of the non-convex nature of our model, we are able to perform a rigorous theoretical analysis of its convergence behavior. Our analysis reveals an interesting dichotomy: a GAN with an optimal discriminator provably converges, while first order approximations of the discriminator steps lead to unstable GAN dynamics and mode collapse. Our result suggests that using first order discriminator steps (the de-facto standard in most existing GAN setups) might be one of the factors that makes GAN training challenging in practice.

18 pages, 4 figures, accepted to ICML 2018

Cited by in corpus (18)

• ContraGAN: Contrastive Learning for Conditional Image Generation
• Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting
• A Systematic Survey of Regularization and Normalization in GANs
• Competitive Gradient Descent
• Convergence Problems with Generative Adversarial Networks (GANs)
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• Mode Collapse and Regularity of Optimal Transportation Maps
• Encoding Invariances in Deep Generative Models
• Smoothness and Stability in GANs
• Relaxed Wasserstein with Applications to GANs
• Lipizzaner: A System That Scales Robust Generative Adversarial Network Training
• The Inductive Bias of Restricted f-GANs
• Training Generative Adversarial Networks via Primal-Dual Subgradient Methods: A Lagrangian Perspective on GAN
• Out-of-Sample Extrapolation with Neuron Editing
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Semi-Implicit Generative Model
• Adaptive Divergence for Rapid Adversarial Optimization
• On the Optimization Landscape of Maximum Mean Discrepancy