Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples

Abstract

We introduce a simple (one line of code) modification to the Generative Adversarial Network (GAN) training algorithm that materially improves results with no increase in computational cost: When updating the generator parameters, we simply zero out the gradient contributions from the elements of the batch that the critic scores as `least realistic'. Through experiments on many different GAN variants, we show that this `top-k update' procedure is a generally applicable improvement. In order to understand the nature of the improvement, we conduct extensive analysis on a simple mixture-of-Gaussians dataset and discover several interesting phenomena. Among these is that, when gradient updates are computed using the worst-scoring batch elements, samples can actually be pushed further away from their nearest mode. We also apply our method to recent GAN variants and improve state-of-the-art FID for conditional generation from 9.21 to 8.57 on CIFAR-10.

arXiv commentNeurIPS 2020. Samarth Sinha and Zhengli Zhao contributed equally as joint first authors

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Adversarial Feature Learning
• Efficient GAN-Based Anomaly Detection
• Generating Natural Adversarial Examples
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• A Variational Inequality Perspective on Generative Adversarial Networks
• Image Augmentations for GAN Training
• Smooth Loss Functions for Deep Top-k Classification
• On GANs and GMMs
• Maximum Entropy Generators for Energy-Based Models
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Prescribed Generative Adversarial Networks
• Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis
• Towards GAN Benchmarks Which Require Generalization
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Improved Consistency Regularization for GANs
• Skill Rating for Generative Models

Cited by in corpus (4)

• Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos
• Sample weighting as an explanation for mode collapse in generative adversarial networks
• MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
• Unrealistic Feature Suppression for Generative Adversarial Networks