Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions

Abstract

To measure the difference between two probability distributions, referred to as the source and target, respectively, we exploit both the chain rule and Bayes' theorem to construct conditional transport (CT), which is constituted by both a forward component and a backward one. The forward CT is the expected cost of moving a source data point to a target one, with their joint distribution defined by the product of the source probability density function (PDF) and a source-dependent conditional distribution, which is related to the target PDF via Bayes' theorem. The backward CT is defined by reversing the direction. The CT cost can be approximated by replacing the source and target PDFs with their discrete empirical distributions supported on mini-batches, making it amenable to implicit distributions and stochastic gradient descent-based optimization. When applied to train a generative model, CT is shown to strike a good balance between mode-covering and mode-seeking behaviors and strongly resist mode collapse. On a wide variety of benchmark datasets for generative modeling, substituting the default statistical distance of an existing generative adversarial network with CT is shown to consistently improve the performance. PyTorch code is provided.

arXiv commentNeurIPS 2021

References in corpus (22)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Large Scale Adversarial Representation Learning
• Generative Moment Matching Networks
• InfoVAE: Information Maximizing Variational Autoencoders
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Variational Lossy Autoencoder
• On Convergence and Stability of GANs
• Learning in Implicit Generative Models
• Image Augmentations for GAN Training
• Variational Inference using Implicit Distributions
• How Well Do WGANs Estimate the Wasserstein Metric?
• Degeneration in VAE: in the Light of Fisher Information Loss
• Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)
• Entropic GANs meet VAEs: A Statistical Approach to Compute Sample Likelihoods in GANs