Small-GAN: Speeding Up GAN Training Using Core-sets
arXiv:1910.13540
Abstract
Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.
References in corpus (5)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- How transferable are features in deep neural networks?
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- The Cramer Distance as a Solution to Biased Wasserstein Gradients
- Maximum Entropy Generators for Energy-Based Models