Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks

Abstract

We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.

arXiv commentPublished in ICLR 2018

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Improved Training of Wasserstein GANs
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Invertible Conditional GANs for image editing
• Generative Moment Matching Networks
• Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
• McGan: Mean and Covariance Feature Matching GAN
• Attribute-Guided Face Generation Using Conditional CycleGAN
• Hierarchical Memory Networks for Answer Selection on Unknown Words

Cited by in corpus (8)

• Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
• Feature Quantization Improves GAN Training
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• Unsupervised Online Learning for Robotic Interestingness with Visual Memory
• Self-Supervised GAN to Counter Forgetting
• Self-Supervised GANs via Auxiliary Rotation Loss
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• OLED: One-Class Learned Encoder-Decoder Network with Adversarial Context Masking for Novelty Detection