Geometric Dataset Distances via Optimal Transport

Abstract

The notion of task similarity is at the core of various machine learning paradigms, such as domain adaptation and meta-learning. Current methods to quantify it are often heuristic, make strong assumptions on the label sets across the tasks, and many are architecture-dependent, relying on task-specific optimal parameters (e.g., require training a model on each dataset). In this work we propose an alternative notion of distance between datasets that (i) is model-agnostic, (ii) does not involve training, (iii) can compare datasets even if their label sets are completely disjoint and (iv) has solid theoretical footing. This distance relies on optimal transport, which provides it with rich geometry awareness, interpretable correspondences and well-understood properties. Our results show that this novel distance provides meaningful comparison of datasets, and correlates well with transfer learning hardness across various experimental settings and datasets.

References in corpus (11)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Domain Adaptation: Learning Bounds and Algorithms
• Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration
• Sample Complexity of Sinkhorn divergences
• Hierarchical Optimal Transport for Document Representation
• Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions
• Adaptive Gradient-Based Meta-Learning Methods
• Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning
• Learning Embeddings into Entropic Wasserstein Spaces
• Dynamics and Reachability of Learning Tasks
• Structured Optimal Transport

Cited by in corpus (8)

• Probing transfer learning with a model of synthetic correlated datasets
• Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Multi-Source Domain Adaptation through Dataset Dictionary Learning in Wasserstein Space
• How to Estimate Model Transferability of Pre-Trained Speech Models?
• Ranking Neural Checkpoints
• Dynamic Flows on Curved Space Generated by Labeled Data
• A Novel Skip Orthogonal List for Dynamic Optimal Transport Problem