On the regularization of Wasserstein GANs

Abstract

Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.

Published as a conference paper at ICLR 2018. * Henning Petzka and Asja Fischer contributed equally to this work (11 pages +13 pages appendix)

References in corpus (7)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improved Training of Wasserstein GANs
• Improved Techniques for Training GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Smooth and Sparse Optimal Transport
• Regularized Optimal Transport and the Rot Mover's Distance

Cited by in corpus (58)

• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Multi-Modal Self-Supervised Learning for Recommendation
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• Banach Wasserstein GAN
• Learning quantum data with the quantum Earth Mover's distance
• A Systematic Survey of Regularization and Normalization in GANs
• Lipschitz Generative Adversarial Nets
• Optimal transport mapping via input convex neural networks
• Understanding GANs: the LQG Setting
• Text to Image Synthesis Using Generative Adversarial Networks
• Exactly Computing the Local Lipschitz Constant of ReLU Networks
• DeepFlow: History Matching in the Space of Deep Generative Models
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Adversarial Lipschitz Regularization
• Some Theoretical Insights into Wasserstein GANs
• An error analysis of generative adversarial networks for learning distributions
• Disentangled Recurrent Wasserstein Autoencoder
• Adversarial Computation of Optimal Transport Maps
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• PA-GAN: Progressive Attention Generative Adversarial Network for Facial Attribute Editing
• Conditioning of three-dimensional generative adversarial networks for pore and reservoir-scale models
• Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?
• Multilevel Optimal Transport: a Fast Approximation of Wasserstein-1 distances
• GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
• Learning disconnected manifolds: a no GANs land
• An Improved Self-supervised GAN via Adversarial Training
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Sinkhorn Distributionally Robust Optimization
• Self-Supervised GANs with Label Augmentation
• Generative Modeling with Optimal Transport Maps
• First Order Generative Adversarial Networks
• Gradient penalty from a maximum margin perspective
• On the Existence of Optimal Transport Gradient for Learning Generative Models
• Wasserstein GANs with Gradient Penalty Compute Congested Transport
• Training Wasserstein GANs without gradient penalties
• Alleviation of Gradient Exploding in GANs: Fake Can Be Real
• Enhancing Mixup-based Semi-Supervised Learning with Explicit Lipschitz Regularization
• Lipschitz neural networks are dense in the set of all Lipschitz functions
• Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
• General Probabilistic Surface Optimization and Log Density Estimation
• Orthogonal Wasserstein GANs
• Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution
• On reproduction of On the regularization of Wasserstein GANs
• Guiding the One-to-one Mapping in CycleGAN via Optimal Transport
• Language Modeling with Generative Adversarial Networks
• Recurrent Adversarial Service Times
• Input Invex Neural Network
• Adversarial Synthesis of Human Pose from Text
• Trust the Critics: Generatorless and Multipurpose WGANs with Initial Convergence Guarantees
• Generating Natural Adversarial Hyperspectral examples with a modified Wasserstein GAN
• Sinusoidal wave generating network based on adversarial learning and its application: synthesizing frog sounds for data augmentation
• Stein Latent Optimization for Generative Adversarial Networks
• Composition and decomposition of GANs
• A Non-linear Differential CNN-Rendering Module for 3D Data Enhancement
• Moreau-Yosida $f$-divergences