A Non-Parametric Test to Detect Data-Copying in Generative Models
arXiv:2004.05675
Abstract
Detecting overfitting in generative models is an important challenge in machine learning. In this work, we formalize a form of overfitting that we call {\em{data-copying}} -- where the generative model memorizes and outputs training samples or small variations thereof. We provide a three sample non-parametric test for detecting data-copying that uses the training set, a separate sample from the target distribution, and a generated sample from the model, and study the performance of our test on several canonical models and datasets. For code \& examples, visit https://github.com/casey-meehan/data-copying
To be published in AISTATS 2020
References in corpus (10)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
- An empirical study on evaluation metrics of generative adversarial networks
- Assessing Generative Models via Precision and Recall
- A Test of Relative Similarity For Model Selection in Generative Models
- Towards GAN Benchmarks Which Require Generalization
- On the Quantitative Analysis of Decoder-Based Generative Models
Cited by in corpus (7)
- Privacy in Deep Learning: A Survey
- On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
- How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
- Understanding Instance-based Interpretability of Variational Auto-Encoders
- Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data
- Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver
- Toward a Generalization Metric for Deep Generative Models