On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
arXiv:2106.03062 · doi:10.1145/3447548.3467198
Abstract
Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric. In this work, we critically evaluate the gameability of these metrics by designing and deploying a generative modeling competition. Our competition received over 11000 submitted models. The competitiveness between participants allowed us to investigate both intentional and unintentional memorization in generative modeling. To detect intentional memorization, we propose the ``Memorization-Informed Fréchet Inception Distance'' (MiFID) as a new memorization-aware metric and design benchmark procedures to ensure that winning submissions made genuine improvements in perceptual quality. Furthermore, we manually inspect the code for the 1000 top-performing models to understand and label different forms of memorization. Our analysis reveals that unintentional memorization is a serious and common issue in popular generative models. The generated images and our memorization labels of those models as well as code to compute MiFID are released to facilitate future studies on benchmarking generative models.
arXiv commentIn Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), August 2021
References in corpus (19)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Neural Architecture Search with Reinforcement Learning
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- NICE: Non-linear Independent Components Estimation
- Improved Techniques for Training GANs
- Adversarial Feature Learning
- Adversarially Learned Inference
- A note on the evaluation of generative models
- A Note on the Inception Score
- Glow: Generative Flow with Invertible 1x1 Convolutions
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Generating Diverse High-Fidelity Images with VQ-VAE-2
- A Non-Parametric Test to Detect Data-Copying in Generative Models
- Semi-Supervised Learning with Normalizing Flows