Reducing Noise in GAN Training with Variance Reduced Extragradient

Abstract

We study the effect of the stochastic gradient noise on the training of generative adversarial networks (GANs) and show that it can prevent the convergence of standard game optimization methods, while the batch version converges. We address this issue with a novel stochastic variance-reduced extragradient (SVRE) optimization algorithm, which for a large class of games improves upon the previous convergence rates proposed in the literature. We observe empirically that SVRE performs similarly to a batch method on MNIST while being computationally cheaper, and that SVRE yields more stable GAN training on standard datasets.

arXiv commentlatest NeurIPS'19 version

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Residual Learning for Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Rethinking the Inception Architecture for Computer Vision
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• No More Pesky Learning Rates
• The Numerics of GANs
• A Variational Inequality Perspective on Generative Adversarial Networks
• Measuring the Effects of Data Parallelism on Neural Network Training
• Improved asynchronous parallel optimization analysis for stochastic incremental methods
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• SMART: The Stochastic Monotone Aggregated Root-Finding Algorithm

Cited by in corpus (32)

• First Analysis of Local GD on Heterogeneous Data
• Near-Optimal Algorithms for Minimax Optimization
• Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• A Decentralized Parallel Algorithm for Training Generative Adversarial Nets
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling
• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
• On the convergence of single-call stochastic extra-gradient methods
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets
• Stochastic Variance Reduction for Variational Inequality Methods
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
• Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems
• Revisiting Stochastic Extragradient
• Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey
• The limits of min-max optimization algorithms: convergence to spurious non-critical sets
• Lower Complexity Bounds of Finite-Sum Optimization Problems: The Results and Construction
• Tight last-iterate convergence rates for no-regret learning in multi-player games
• Extragradient Method: $O(1/K)$ Last-Iterate Convergence for Monotone Variational Inequalities and Connections With Cocoercivity
• Gradient Free Minimax Optimization: Variance Reduction and Faster Convergence
• A Unified Analysis of Variational Inequality Methods: Variance Reduction, Sampling, Quantization and Coordinate Descent
• Taming GANs with Lookahead-Minmax
• Extragradient with player sampling for faster Nash equilibrium finding
• Near Optimal Stochastic Algorithms for Finite-Sum Unbalanced Convex-Concave Minimax Optimization
• Stochastic Extragradient: General Analysis and Improved Rates
• CDMA: A Practical Cross-Device Federated Learning Algorithm for General Minimax Problems
• Extrapolation for Large-batch Training in Deep Learning
• Efficient Projection-Free Algorithms for Saddle Point Problems
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Tessellated Wasserstein Auto-Encoders
• SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities
• Complex Momentum for Optimization in Games