Resampling Base Distributions of Normalizing Flows
arXiv:2110.15828
Abstract
Normalizing flows are a popular class of models for approximating probability distributions. However, their invertible nature limits their ability to model target distributions whose support have a complex topological structure, such as Boltzmann distributions. Several procedures have been proposed to solve this problem but many of them sacrifice invertibility and, thereby, tractability of the log-likelihood as well as other desirable properties. To address these limitations, we introduce a base distribution for normalizing flows based on learned rejection sampling, allowing the resulting normalizing flow to model complicated distributions without giving up bijectivity. Furthermore, we develop suitable learning algorithms using both maximizing the log-likelihood and the optimization of the Kullback-Leibler divergence, and apply them to various sample problems, i.e. approximating 2D densities, density estimation of tabular data, image generation, and modeling Boltzmann distributions. In these experiments our method is competitive with or outperforms the baselines.
References in corpus (12)
- NICE: Non-linear Independent Components Estimation
- Generating Long Sequences with Sparse Transformers
- Variational Diffusion Models
- The Reversible Residual Network: Backpropagation Without Storing Activations
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- PixelSNAIL: An Improved Autoregressive Generative Model
- SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
- Stabilizing Invertible Neural Networks Using Mixture Models
- Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
- Semi-Supervised Learning with Normalizing Flows
- Smooth Normalizing Flows
- Normalizing Flows with Multi-Scale Autoregressive Priors