Implicit Normalizing Flows

Abstract

Normalizing flows define a probability distribution by an explicit invertible transformation $\mathbf{z}=f(\mathbf{x})$. In this work, we present implicit normalizing flows (ImpFlows), which generalize normalizing flows by allowing the mapping to be implicitly defined by the roots of an equation $F(\mathbf{z},\mathbf{x})=0$. ImpFlows build on residual flows (ResFlows) with a proper balance between expressiveness and tractability. Through theoretical analysis, we show that the function space of ImpFlow is strictly richer than that of ResFlows. Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has a non-negligible approximation error. However, the function is exactly representable by a single-block ImpFlow. We propose a scalable algorithm to train and draw samples from ImpFlows. Empirically, we evaluate ImpFlow on several classification and density modeling tasks, and ImpFlow outperforms ResFlow with a comparable amount of parameters on all the benchmarks.

References in corpus (7)

• Variational Inference with Normalizing Flows
• Deep Equilibrium Models
• Residual Flows for Invertible Generative Modeling
• Emerging Convolutions for Generative Normalizing Flows
• Continuous-Time Flows for Efficient Inference and Density Estimation
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Robust learning with implicit residual networks

Cited by in corpus (5)

• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• On Training Implicit Models
• Can convolutional ResNets approximately preserve input distances? A frequency analysis perspective
• Rectangular Flows for Manifold Learning
• Viscos Flows: Variational Schur Conditional Sampling With Normalizing Flows