Sliced Iterative Normalizing Flows

Abstract

We develop an iterative (greedy) deep learning (DL) algorithm which is able to transform an arbitrary probability distribution function (PDF) into the target PDF. The model is based on iterative Optimal Transport of a series of 1D slices, matching on each slice the marginal PDF to the target. The axes of the orthogonal slices are chosen to maximize the PDF difference using Wasserstein distance at each iteration, which enables the algorithm to scale well to high dimensions. As special cases of this algorithm, we introduce two sliced iterative Normalizing Flow (SINF) models, which map from the data to the latent space (GIS) and vice versa (SIG). We show that SIG is able to generate high quality samples of image datasets, which match the GAN benchmarks, while GIS obtains competitive results on density estimation tasks compared to the density trained NFs, and is more stable, faster, and achieves higher $p(x)$ when trained on small training sets. SINF approach deviates significantly from the current DL paradigm, as it is greedy and does not use concepts such as mini-batching, stochastic gradient descent and gradient back-propagation through deep layers.

arXiv comment19 pages, 12 figures, 7 tables. Code available at https://github.com/biweidai/SINF

References in corpus (25)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Wasserstein Auto-Encoders
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Iterative Gaussianization: from ICA to Random Rotations
• The Radon cumulative distribution transform and its application to image classification
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Stabilizing Generative Adversarial Networks: A Survey
• High-Dimensional Probability Estimation with Deep Density Models
• Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
• Subspace Robust Wasserstein Distances
• Probabilistic Autoencoder
• Statistical and Topological Properties of Sliced Probability Divergences
• Generative Latent Flow
• Generalized Sliced Wasserstein Distances
• Cramer-Wold AutoEncoder
• Orthogonal Estimation of Wasserstein Distances
• Gaussianization Flows
• Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions

Cited by in corpus (2)

• The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
• Unsupervised in-distribution anomaly detection of new physics through conditional density estimation