Sinkhorn AutoEncoders

Abstract

Optimal transport offers an alternative to maximum likelihood for learning generative autoencoding models. We show that minimizing the p-Wasserstein distance between the generator and the true data distribution is equivalent to the unconstrained min-min optimization of the p-Wasserstein distance between the encoder aggregated posterior and the prior in latent space, plus a reconstruction error. We also identify the role of its trade-off hyperparameter as the capacity of the generator: its Lipschitz constant. Moreover, we prove that optimizing the encoder over any class of universal approximators, such as deterministic neural networks, is enough to come arbitrarily close to the optimum. We therefore advertise this framework, which holds for any metric space and prior, as a sweet-spot of current generative autoencoding objectives. We then introduce the Sinkhorn auto-encoder (SAE), which approximates and minimizes the p-Wasserstein distance in latent space via backprogation through the Sinkhorn algorithm. SAE directly works on samples, i.e. it models the aggregated posterior as an implicit distribution, with no need for a reparameterization trick for gradients estimations. SAE is thus able to work with different metric spaces and priors with minimal adaptations. We demonstrate the flexibility of SAE on latent spaces with different geometries and priors and compare with other methods on benchmark data sets.

arXiv commentAccepted for oral presentation at UAI19

References in corpus (6)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Learning with a Wasserstein Loss
• Unsupervised Learning by Predicting Noise
• Implicit Reparameterization Gradients
• Sample Complexity of Sinkhorn divergences
• Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance

Cited by in corpus (15)

• Machine Learning for Anomaly Detection in Particle Physics
• Learning to Simulate High Energy Particle Collisions from Unlabeled Data
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Minibatch optimal transport distances; analysis and applications
• On the extremal points of the ball of the Benamou-Brenier energy
• Entropic regularization of Wasserstein distance between infinite-dimensional Gaussian measures and Gaussian processes
• Neural Topic Model via Optimal Transport
• Learning Implicit Generative Models with Theoretical Guarantees
• Augmented Sliced Wasserstein Distances
• Investigating and Improving Latent Density Segmentation Models for Aleatoric Uncertainty Quantification in Medical Imaging
• Generative Learning With Euler Particle Transport
• On the Latent Holes of VAEs for Text Generation
• Spherical Sliced-Wasserstein
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
• WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding