A Test of Relative Similarity For Model Selection in Generative Models

Abstract

Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches. Model selection in this generative setting can be challenging, however, particularly when likelihoods are not easily accessible. To address this issue, we introduce a statistical test of relative similarity, which is used to determine which of two models generates samples that are significantly closer to a real-world reference dataset of interest. We use as our test statistic the difference in maximum mean discrepancies (MMDs) between the reference dataset and each model dataset, and derive a powerful, low-variance test based on the joint asymptotic distribution of the MMDs between each reference-model pair. In experiments on deep generative models, including the variational auto-encoder and generative moment matching network, the tests provide a meaningful ranking of model performance as a function of parameter and training settings.

International Conference on Learning Representations 2016

Cited by in corpus (22)

• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• An empirical study on evaluation metrics of generative adversarial networks
• Demystifying MMD GANs
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• On gradient regularizers for MMD GANs
• A Design Methodology for Efficient Implementation of Deconvolutional Neural Networks on an FPGA
• Physics-informed GANs for Coastal Flood Visualization
• TG-GAN: Continuous-time Temporal Graph Generation with Deep Generative Models
• Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning
• A Non-Parametric Test to Detect Data-Copying in Generative Models
• Unbiased estimators for the variance of MMD estimators
• Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions
• Testing Goodness of Fit of Conditional Density Models with Kernels
• Concept Drift Detection: Dealing with MissingValues via Fuzzy Distance Estimations
• KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling
• Amortised Learning by Wake-Sleep
• Fast Non-Parametric Tests of Relative Dependency and Similarity
• Goodness-of-Fit Test for Mismatched Self-Exciting Processes
• Implicit Generative Copulas