Gradient descent GAN optimization is locally stable

Abstract

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still \emph{locally asymptotically stable} for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which \emph{is} able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.

Cited by in corpus (25)

• Is Generator Conditioning Causally Related to GAN Performance?
• A Convex Duality Framework for GANs
• BourGAN: Generative Networks with Metric Embeddings
• Generative Adversarial Network Training is a Continual Learning Problem
• Understanding GANs: the LQG Setting
• FedGAN: Federated Generative Adversarial Networks for Distributed Data
• Linear Convergence of the Primal-Dual Gradient Method for Convex-Concave Saddle Point Problems without Strong Convexity
• Towards a Better Understanding and Regularization of GAN Training Dynamics
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• The limits of min-max optimization algorithms: convergence to spurious non-critical sets
• The Inductive Bias of Restricted f-GANs
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• Dynamics of Fourier Modes in Torus Generative Adversarial Networks
• MMGAN: Generative Adversarial Networks for Multi-Modal Distributions
• Saddle Point Optimization with Approximate Minimization Oracle
• MR-GAN: Manifold Regularized Generative Adversarial Networks
• First Order Generative Adversarial Networks
• K-Beam Minimax: Efficient Optimization for Deep Adversarial Learning
• A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
• Generalized Energy Based Models
• Machine vs Machine: Minimax-Optimal Defense Against Adversarial Examples
• Control of Overfitting with Physics
• Capturing the diversity of biological tuning curves using generative adversarial networks
• Functional Space Analysis of Local GAN Convergence
• Local Stability and Performance of Simple Gradient Penalty mu-Wasserstein GAN