The Unusual Effectiveness of Averaging in GAN Training

Abstract

We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{https://github.com/yasinyazici/EMA_GAN}}

arXiv commentPublished as a conference paper at ICLR 2019

References in corpus (14)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Improved Techniques for Training GANs
• Which Training Methods for GANs do actually Converge?
• A Note on the Inception Score
• The Numerics of GANs
• A Variational Inequality Perspective on Generative Adversarial Networks
• The Mechanics of n-Player Differentiable Games
• Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile
• From Averaging to Acceleration, There is Only a Step-size
• Activation Maximization Generative Adversarial Nets
• Negative Momentum for Improved Game Dynamics
• Fictitious GAN: Training GANs with Historical Models

Cited by in corpus (29)

• Lookahead Optimizer: k steps forward, 1 step back
• On Data Augmentation for GAN Training
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• A Variational Inequality Perspective on Generative Adversarial Networks
• ContraGAN: Contrastive Learning for Conditional Image Generation
• StarGAN v2: Diverse Image Synthesis for Multiple Domains
• The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
• Last-iterate convergence rates for min-max optimization
• A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
• Convolutional Generation of Textured 3D Meshes
• What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction
• Last Iterate is Slower than Averaged Iterate in Smooth Convex-Concave Saddle Point Problems
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• An Improved Self-supervised GAN via Adversarial Training
• Vortices Instead of Equilibria in MinMax Optimization: Chaos and Butterfly Effects of Online Learning in Zero-Sum Games
• Neural SDEs as Infinite-Dimensional GANs
• Increasing Iterate Averaging for Solving Saddle-Point Problems
• Follow-the-Regularized-Leader Routes to Chaos in Routing Games
• Fast and Furious Learning in Zero-Sum Games: Vanishing Regret with Non-Vanishing Step Sizes
• Taming GANs with Lookahead-Minmax
• Data Cleansing for GANs
• Alleviation of Gradient Exploding in GANs: Fake Can Be Real
• Efficient and Accurate Gradients for Neural SDEs
• A stochastic variant of replicator dynamics in zero-sum games and its invariant measures
• MR-Contrast-Aware Image-to-Image Translations with Generative Adversarial Networks
• MSGDD-cGAN: Multi-Scale Gradients Dual Discriminator Conditional Generative Adversarial Network
• Venn GAN: Discovering Commonalities and Particularities of Multiple Distributions
• Dist-GAN: An Improved GAN using Distance Constraints
• Tom: Leveraging trend of the observed gradients for faster convergence