The Cramer Distance as a Solution to Biased Wasserstein Gradients

Abstract

The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling. In this paper we describe three natural properties of probability divergences that reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting that this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramér distance. We show that the Cramér distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. To illustrate the relevance of the Cramér distance in practice we design a new algorithm, the Cramér Generative Adversarial Network (GAN), and show that it performs significantly better than the related Wasserstein GAN.

References in corpus (6)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improved Training of Wasserstein GANs
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Comparison of Maximum Likelihood and GAN-based training of Real NVPs

Cited by in corpus (91)

• Deep Neural Networks and Tabular Data: A Survey
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• How Generative Adversarial Networks and Their Variants Work: An Overview
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• A Distributional Perspective on Reinforcement Learning
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• Demystifying MMD GANs
• A Probabilistic U-Net for Segmentation of Ambiguous Images
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• On the regularization of Wasserstein GANs
• ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity?
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Controlling Physical Attributes in GAN-Accelerated Simulation of Electromagnetic Calorimeters
• Stabilizing Generative Adversarial Networks: A Survey
• CTAB-GAN: Effective Table Data Synthesizing
• On gradient regularizers for MMD GANs
• Maximum Mean Discrepancy Gradient Flow
• Is Generator Conditioning Causally Related to GAN Performance?
• An Analysis of Categorical Distributional Reinforcement Learning
• Improving GANs Using Optimal Transport
• Lipschitz Generative Adversarial Nets
• Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• Unbalanced minibatch Optimal Transport; applications to Domain Adaptation
• Small-GAN: Speeding Up GAN Training Using Core-sets
• Autoregressive Quantile Networks for Generative Modeling
• Airline Passenger Name Record Generation using Generative Adversarial Networks
• Cherenkov Detectors Fast Simulation Using Neural Networks
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Learning to Generate Novel Domains for Domain Generalization
• Learning Generative Models across Incomparable Spaces
• Black-Box Optimization with Local Generative Surrogates
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• Statistical and Topological Properties of Sliced Probability Divergences
• Learning with minibatch Wasserstein : asymptotic and gradient properties
• Minibatch optimal transport distances; analysis and applications
• A Survey on Optimal Transport for Machine Learning: Theory and Applications
• Towards Robust, Locally Linear Deep Networks
• Disentangled Recurrent Wasserstein Autoencoder
• The Six Fronts of the Generative Adversarial Networks
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
• Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
• GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection
• Quantitatively Evaluating GANs With Divergences Proposed for Training
• On Wasserstein Reinforcement Learning and the Fokker-Planck equation
• ChildGAN: Large Scale Synthetic Child Facial Data Using Domain Adaptation in StyleGAN
• Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images
• Training Generative Adversarial Networks Via Turing Test
• Learning deep kernels for exponential family densities
• Likelihood Estimation for Generative Adversarial Networks
• Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark
• How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals
• Towards Robust GAN-generated Image Detection: a Multi-view Completion Representation
• Estimating Barycenters of Measures in High Dimensions
• Training Generative Reversible Networks
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors
• GANs beyond divergence minimization
• Metric Learning-based Generative Adversarial Network
• Learning Implicit Generative Models with the Method of Learned Moments
• FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction
• Implicit Distributional Reinforcement Learning
• First Order Generative Adversarial Networks
• PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning
• On Transportation of Mini-batches: A Hierarchical Approach
• A gradual, semi-discrete approach to generative network training via explicit Wasserstein minimization
• MMGAN: Manifold Matching Generative Adversarial Network
• Dodging DeepFake Detection via Implicit Spatial-Domain Notch Filtering
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Reciprocal Adversarial Learning via Characteristic Functions
• Unsupervised Hierarchy Matching with Optimal Transport over Hyperbolic Spaces
• Generalized Sliced Distances for Probability Distributions
• Correcting Nuisance Variation using Wasserstein Distance
• PC-GAIN: Pseudo-label Conditional Generative Adversarial Imputation Networks for Incomplete Data
• Radon Sobolev Variational Auto-Encoders
• Blur, Noise, and Compression Robust Generative Adversarial Networks
• RSD-GAN: Regularized Sobolev Defense GAN Against Speech-to-Text Adversarial Attacks
• Alignment Attention by Matching Key and Query Distributions
• Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions
• Post Selection Inference with Incomplete Maximum Mean Discrepancy Estimator
• Restricting Greed in Training of Generative Adversarial Network
• Implicit Manifold Learning on Generative Adversarial Networks
• Complexity Controlled Generative Adversarial Networks
• Adaptive Divergence for Rapid Adversarial Optimization
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Noise Robust Generative Adversarial Networks
• Towards GANs' Approximation Ability
• SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Dynamic Flows on Curved Space Generated by Labeled Data