A domain agnostic measure for monitoring and evaluating GANs

Abstract

Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be determined by simply inspecting the generator and discriminator loss curves as they behave non-intuitively. We leverage the notion of duality gap from game theory to propose a measure that addresses both (i) and (ii) at a low computational cost. Extensive experiments show the effectiveness of this measure to rank different GAN models and capture the typical GAN failure scenarios, including mode collapse and non-convergent behaviours. This evaluation metric also provides meaningful monitoring on the progression of the loss during training. It highly correlates with FID on natural image datasets, and with domain specific scores for text, sound and cosmology data where FID is not directly suitable. In particular, our proposed metric requires no labels or a pretrained classifier, making it domain agnostic.

Cited by in corpus (10)

• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
• Mimicry: Towards the Reproducibility of GAN Research
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Generative Minimization Networks: Training GANs Without Competition
• On Characterizing GAN Convergence Through Proximal Duality Gap
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds
• Conformalised data synthesis
• Fuzzy Generative Adversarial Networks
• Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence