On the Convergence and Robustness of Training GANs with Regularized Optimal Transport

Abstract

Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.

Cited by in corpus (38)

• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applications
• Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advances
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• GANs May Have No Nash Equilibria
• Smoothness and Stability in GANs
• Min-Max Optimization without Gradients: Convergence and Applications to Adversarial ML
• Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities
• Refinement of molecular dynamics ensembles using experimental data and flexible forward models
• Rényi Fair Inference
• Improving the Speed and Quality of GAN by Adversarial Training
• Generalization Properties of Optimal Transport GANs with Latent Distribution Learning
• Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark
• Online and Bandit Algorithms for Nonstationary Stochastic Saddle-Point Optimization
• The Complexity of Nonconvex-Strongly-Concave Minimax Optimization
• Estimating Barycenters of Measures in High Dimensions
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• Train simultaneously, generalize better: Stability of gradient-based minimax learners
• Generative Modeling with Optimal Transport Maps
• Nonconvex-Nonconcave Min-Max Optimization with a Small Maximization Domain
• Generalized Energy Based Models
• Optimal Transport Relaxations with Application to Wasserstein GANs
• Finding Second-Order Stationary Points in Nonconvex-Strongly-Concave Minimax Optimization
• Frank-Wolfe Methods in Probability Space
• On the Existence of Optimal Transport Gradient for Learning Generative Models
• Deconstructing Generative Adversarial Networks
• Randomized Stochastic Gradient Descent Ascent
• Semi-Anchored Multi-Step Gradient Descent Ascent Method for Structured Nonconvex-Nonconcave Composite Minimax Problems
• Symmetric Wasserstein Autoencoders
• Trust the Critics: Generatorless and Multipurpose WGANs with Initial Convergence Guarantees
• A Wasserstein Minimax Framework for Mixed Linear Regression
• Understanding Entropic Regularization in GANs
• Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascent
• Direct-Search for a Class of Stochastic Min-Max Problems
• Informative GANs via Structured Regularization of Optimal Transport
• Generative Adversarial Network: Some Analytical Perspectives
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points
• Convex-Concave Min-Max Stackelberg Games