Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
arXiv:2312.04078 · doi:10.1214/24-SS149
Abstract
Quantifying the similarity between datasets has widespread applications in statistics and machine learning. The performance of a predictive model on novel datasets, referred to as generalizability, depends on how similar the training and evaluation datasets are. Exploiting or transferring insights between similar datasets is a key aspect of meta-learning and transfer-learning. In simulation studies, the similarity between distributions of simulated datasets and real datasets, for which the performance of methods is assessed, is crucial. In two- or -sample testing, it is checked, whether the underlying distributions of two or more datasets coincide. Extremely many approaches for quantifying dataset similarity have been proposed in the literature. We examine more than 100 methods and provide a taxonomy, classifying them into ten classes. In an extensive review of these methods the main underlying ideas, formal definitions, and important properties are introduced. We compare the 118 methods in terms of their applicability, interpretability, and theoretical properties, in order to provide recommendations for selecting an appropriate dataset similarity measure based on the specific goal of the dataset comparison and on the properties of the datasets at hand. An online tool facilitates the choice of the appropriate dataset similarity measure.
124 pages
References in corpus (43)
- Rényi Divergence and Kullback-Leibler Divergence
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Equivalence of distance-based and RKHS-based statistics in hypothesis testing
- Kernel Mean Embedding of Distributions: A Review and Beyond
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
- -divergence Inequalities
- Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
- DISCO analysis: A nonparametric extension of analysis of variance
- A new graph-based two-sample test for multivariate and object data
- Learning Deep Kernels for Non-Parametric Two-Sample Tests
- Learning Generative Models with Sinkhorn Divergences
- Geometric Dataset Distances via Optimal Transport
- Revisiting Classifier Two-Sample Tests
- On perfect clustering of high dimension, low sample size data
- Two-sample Bayesian Nonparametric Hypothesis Testing
- A weighted edge-count two-sample test for multivariate and object data
- FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test
- Quadratic distances on probabilities: A unified foundation
- Kernel Distribution Embeddings: Universal Kernels, Characteristic Kernels and Kernel Metrics on Distributions
- Fully distribution-free center-outward rank tests for multiple-output regression and MANOVA
- Distances between Data Sets Based on Summary Statistics
- Interpretable Distribution Features with Maximum Testing Power
- Two-sample Test using Projected Wasserstein Distance
- Bayesian Learning of Kernel Embeddings
- Better-Than-Chance Classification for Signal Detection
- Multivariate Brenier cumulative distribution functions and their application to non-parametric testing
- Generalized Kernel Two-Sample Tests
- Informative Features for Model Comparison
- Ball: An R package for detecting distribution difference and association in metric spaces
- Testing Hypotheses by Regularized Maximum Mean Discrepancy
- On high-dimensional modifications of some graph-based two-sample tests
- Classification Logit Two-sample Testing by Neural Networks
- An Efficient and Distribution-Free Two-Sample Test Based on Energy Statistics and Random Projections
- A new test for the multivariate two-sample problem based on the concept of minimum energy
- Two-sample Testing Using Deep Learning
- Two-sample Test with Kernel Projected Wasserstein Distance
- Graph-Based Two-Sample Tests for Data with Repeated Observations
- Multivariate Ranks and Quantiles using Optimal Transport: Consistency, Rates, and Nonparametric Testing
- Pitman Efficiency Lower Bounds for Multivariate Distribution-Free Tests Based on Optimal Transport
- Limiting distributions of graph-based test statistics on sparse and dense graphs
- New graph-based multi-sample tests for high-dimensional and non-Euclidean data
- A new ranking scheme for modern data and its application to two-sample hypothesis testing