Implicit competitive regularization in GANs

Abstract

To improve the stability of GAN training we need to understand why they can produce realistic samples. Presently, this is attributed to properties of the divergence obtained under an optimal discriminator. This argument has a fundamental flaw: If we do not impose regularity of the discriminator, it can exploit visually imperceptible errors of the generator to always achieve the maximal generator loss. In practice, gradient penalties are used to regularize the discriminator. However, this needs a metric on the space of images that captures visual similarity. Such a metric is not known, which explains the limited success of gradient penalties in stabilizing GANs. We argue that the performance of GANs is instead due to the implicit competitive regularization (ICR) arising from the simultaneous optimization of generator and discriminator. ICR promotes solutions that look real to the discriminator and thus leverages its inductive biases to generate realistic images. We show that opponent-aware modelling of generator and discriminator, as present in competitive gradient descent (CGD), can significantly strengthen ICR and thus stabilize GAN training without explicit regularization. In our experiments, we use an existing implementation of WGAN-GP and show that by training it with CGD we can improve the inception score (IS) on CIFAR10 for a wide range of scenarios, without any hyperparameter tuning. The highest IS is obtained by combining CGD with the WGAN-loss, without any explicit regularization.

arXiv commentThe code used to produce the numerical experiments can be found under http://github.com/devzhk/ICR . A high-level overview of this work can be found under https://f-t-s.github.io/projects/icr/

References in corpus (26)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• On Calibration of Modern Neural Networks
• Understanding deep learning requires rethinking generalization
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Generative Moment Matching Networks
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• On Convergence and Stability of GANs
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• The Numerics of GANs
• Demystifying MMD GANs
• The Mechanics of n-Player Differentiable Games
• Banach Wasserstein GAN
• On integral probability metrics, ϕ-divergences and binary classification
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Matrix Factorization
• What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
• Competitive Gradient Descent
• Convergence of Learning Dynamics in Stackelberg Games
• Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
• GANs May Have No Nash Equilibria
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Implicit Regularization in Over-parameterized Neural Networks
• A gradual, semi-discrete approach to generative network training via explicit Wasserstein minimization

Cited by in corpus (10)

• Stabilizing Generative Adversarial Networks: A Survey
• Graph Generative Adversarial Networks for Sparse Data Generation in High Energy Physics
• A case for new neural network smoothness constraints
• Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
• Generative Minimization Networks: Training GANs Without Competition
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• SDE approximations of GANs training and its long-run behavior
• Competitive Policy Optimization
• Competitive Mirror Descent
• Complex Momentum for Optimization in Games