The Shape of Data: Intrinsic Distance for Data Distributions
arXiv:1905.11141
Abstract
The ability to represent and compare machine learning models is crucial in order to quantify subtle model changes, evaluate generative models, and gather insights on neural network architectures. Existing techniques for comparing data distributions focus on global data properties such as mean and covariance; in that sense, they are extrinsic and uni-scale. We develop a first-of-its-kind intrinsic and multi-scale method for characterizing and comparing data manifolds, using a lower-bound of the spectral variant of the Gromov-Wasserstein inter-manifold distance, which compares all data moments. In a thorough experimental study, we demonstrate that our method effectively discerns the structure of data manifolds even on unaligned data of different dimensionalities; moreover, we showcase its efficacy in evaluating the quality of generative models.
arXiv commentPublished in ICLR'2020
References in corpus (8)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Similarity of Neural Network Representations Revisited
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
- NetLSD: Hearing the Shape of a Graph
- On the Dimensionality of Word Embedding
- Geometry Score: A Method For Comparing Generative Adversarial Networks
- Estimating the Spectral Density of Large Implicit Matrices
Cited by in corpus (6)
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- Just SLaQ When You Approximate: Accurate Spectral Distances for Web-Scale Graphs
- Generalized Spectral Clustering via Gromov-Wasserstein Learning
- Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
- Manifold Topology Divergence: a Framework for Comparing Data Manifolds
- Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions