Informative Features for Model Comparison

Abstract

Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models. We propose two new statistical tests which are nonparametric, computationally efficient (runtime complexity is linear in the sample size), and interpretable. As a unique advantage, our tests can produce a set of examples (informative features) indicating the regions in the data domain where one model fits significantly better than the other. In a real-world problem of comparing GAN models, the test power of our new test matches that of the state-of-the-art test of relative goodness of fit, while being one order of magnitude faster.

arXiv commentAccepted to NIPS 2018

Cited by in corpus (13)

• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Learning deep kernels for exponential family densities
• Learning Kernel Tests Without Data Splitting
• Two-sample Testing Using Deep Learning
• Exhaustive goodness-of-fit via smoothed inference and graphics
• Testing Goodness of Fit of Conditional Density Models with Kernels
• A kernel test for quasi-independence
• A Stein Goodness-of-fit Test for Directional Distributions
• Kernel Mean Matching for Content Addressability of GANs
• A Stein Goodness of fit Test for Exponential Random Graph Models
• Style transfer between Microscopy and Magnetic Resonance Imaging via Generative Adversarial Network in small sample size settings
• More Powerful Selective Kernel Tests for Feature Selection
• Interpretable Stein Goodness-of-fit Tests on Riemannian Manifolds