On integral probability metrics, ϕ-divergences and binary classification

Abstract

A class of distance measures on probabilities -- the integral probability metrics (IPMs) -- is addressed: these include the Wasserstein distance, Dudley metric, and Maximum Mean Discrepancy. IPMs have thus far mostly been used in more abstract settings, for instance as theoretical tools in mass transportation problems, and in metrizing the weak topology on the set of all Borel probability measures defined on a metric space. Practical applications of IPMs are less common, with some exceptions in the kernel machines literature. The present work contributes a number of novel properties of IPMs, which should contribute to making IPMs more widely used in practice, for instance in areas where $\varphi$-divergences are currently popular. First, to understand the relation between IPMs and $\varphi$-divergences, the necessary and sufficient conditions under which these classes intersect are derived: the total variation distance is shown to be the only non-trivial $\varphi$-divergence that is also an IPM. This shows that IPMs are essentially different from $\varphi$-divergences. Second, empirical estimates of several IPMs from finite i.i.d. samples are obtained, and their consistency and convergence rates are analyzed. These estimators are shown to be easily computable, with better rates of convergence than estimators of $\varphi$-divergences. Third, a novel interpretation is provided for IPMs by relating them to binary classification, where it is shown that the IPM between class-conditional distributions is the negative of the optimal risk associated with a binary classifier. In addition, the smoothness of an appropriate binary classifier is proved to be inversely related to the distance between the class-conditional distributions, measured in terms of an IPM.

18 pages

References in corpus (3)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Entropy and information in neural spike trains: Progress on the sampling problem
• Information, Divergence and Risk for Binary Experiments

Cited by in corpus (48)

• Generative Modeling by Estimating Gradients of the Data Distribution
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• MOPO: Model-based Offline Policy Optimization
• Stabilizing Training of Generative Adversarial Networks through Regularization
• Learning in Implicit Generative Models
• An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction
• Demystifying MMD GANs
• The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine
• Stabilizing Generative Adversarial Networks: A Survey
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• GenDICE: Generalized Offline Estimation of Stationary Values
• Learning Weighted Representations for Generalization Across Designs
• Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects
• Advances in Variational Inference
• Statistical and Topological Properties of Sliced Probability Divergences
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Implicit competitive regularization in GANs
• Sliced Kernelized Stein Discrepancy
• Calibration, Entropy Rates, and Memory in Language Models
• Classification and reconstruction of optical quantum states with deep neural networks
• A Continuous Variable Born Machine
• TripletGAN: Training Generative Model with Triplet Loss
• Distance Based Source Domain Selection for Sentiment Classification
• Limit Distribution Theory for the Smooth 1-Wasserstein Distance with Applications
• Representation Balancing MDPs for Off-Policy Policy Evaluation
• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics
• Stable Policy Optimization via Off-Policy Divergence Regularization
• Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Statistical Optimal Transport posed as Learning Kernel Embedding
• Learning Decomposed Representation for Counterfactual Inference
• Neural Tangent Kernel Maximum Mean Discrepancy
• Counterfactual Representation Learning with Balancing Weights
• Moment-Based Domain Adaptation: Learning Bounds and Algorithms
• Variational Inference with Holder Bounds
• RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
• Imitation Learning with Sinkhorn Distances
• Distributional Robustness with IPMs and links to Regularization and GANs
• A Novel Framework for Selection of GANs for an Application
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Semi-Supervised Learning with IPM-based GANs: an Empirical Study
• Parallel Total Variation Distance Estimation with Neural Networks for Merging Over-Clusterings
• Calibration tests beyond classification
• Bridging the Gap Between $f$-GANs and Wasserstein GANs
• An Empirical Evaluation of the Approximation of Subjective Logic Operators Using Monte Carlo Simulations
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data