Online Kernel based Generative Adversarial Networks

Abstract

One of the major breakthroughs in deep learning over the past five years has been the Generative Adversarial Network (GAN), a neural network-based generative model which aims to mimic some underlying distribution given a dataset of samples. In contrast to many supervised problems, where one tries to minimize a simple objective function of the parameters, GAN training is formulated as a min-max problem over a pair of network parameters. While empirically GANs have shown impressive success in several domains, researchers have been puzzled by unusual training behavior, including cycling so-called mode collapse. In this paper, we begin by providing a quantitative method to explore some of the challenges in GAN training, and we show empirically how this relates fundamentally to the parametric nature of the discriminator network. We propose a novel approach that resolves many of these issues by relying on a kernel-based non-parametric discriminator that is highly amenable to online training---we call this the Online Kernel-based Generative Adversarial Networks (OKGAN). We show empirically that OKGANs mitigate a number of training issues, including mode collapse and cycling, and are much more amenable to theoretical guarantees. OKGANs empirically perform dramatically better, with respect to reverse KL-divergence, than other GAN formulations on synthetic data; on classical vision datasets such as MNIST, SVHN, and CelebA, show comparable performance.

References in corpus (15)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Spectral Normalization for Generative Adversarial Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Generative Moment Matching Networks
• On Convergence and Stability of GANs
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• The Numerics of GANs
• Demystifying MMD GANs
• Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
• Approximability of Discriminators Implies Diversity in GANs
• Towards a Better Understanding and Regularization of GAN Training Dynamics
• Theoretical limitations of Encoder-Decoder GAN architectures