Small-GAN: Speeding Up GAN Training Using Core-sets

Abstract

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.

References in corpus (20)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• How transferable are features in deep neural networks?
• Spectral Normalization for Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Efficient GAN-Based Anomaly Detection
• MaskGAN: Better Text Generation via Filling in the______
• Variational Continual Learning
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• A Variational Inequality Perspective on Generative Adversarial Networks
• Measuring the Effects of Data Parallelism on Neural Network Training
• Generative Adversarial Active Learning
• Maximum Entropy Generators for Energy-Based Models
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• Improving GANs Using Optimal Transport
• Skill Rating for Generative Models

Cited by in corpus (10)

• Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
• Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Improved Consistency Regularization for GANs
• ALGAN: Anomaly Detection by Generating Pseudo Anomalous Data via Latent Variables
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Group Equivariant Generative Adversarial Networks
• Instance Selection for GANs
• Accelerating Neural Architecture Search via Proxy Data