SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows

Abstract

Normalizing flows and variational autoencoders are powerful generative models that can represent complicated density functions. However, they both impose constraints on the models: Normalizing flows use bijective transformations to model densities whereas VAEs learn stochastic transformations that are non-invertible and thus typically do not provide tractable estimates of the marginal likelihood. In this paper, we introduce SurVAE Flows: A modular framework of composable transformations that encompasses VAEs and normalizing flows. SurVAE Flows bridge the gap between normalizing flows and VAEs with surjective transformations, wherein the transformations are deterministic in one direction -- thereby allowing exact likelihood computation, and stochastic in the reverse direction -- hence providing a lower bound on the corresponding likelihood. We show that several recently proposed methods, including dequantization and augmented normalizing flows, can be expressed as SurVAE Flows. Finally, we introduce common operations such as the max value, the absolute value, sorting and stochastic permutation as composable layers in SurVAE Flows.

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Denoising Diffusion Probabilistic Models
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• RNADE: The real-valued neural autoregressive density-estimator
• A Deep and Tractable Density Estimator
• Emerging Convolutions for Generative Normalizing Flows
• Discrete Flows: Invertible Generative Models of Discrete Data
• Sum-of-Squares Polynomial Flow
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• A RAD approach to deep mixture models
• BRUNO: A Deep Recurrent Model for Exchangeable Data

Cited by in corpus (12)

• Pre-training Molecular Graph Representation with 3D Geometry
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• normflows: A PyTorch Package for Normalizing Flows
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• A RAD approach to deep mixture models
• Text to Image Synthesis using Stacked Conditional Variational Autoencoders and Conditional Generative Adversarial Networks
• ChartPointFlow for Topology-Aware 3D Point Cloud Generation
• E(n) Equivariant Normalizing Flows
• EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation
• Resampling Base Distributions of Normalizing Flows
• Prior Flow Variational Autoencoder: A density estimation model for Non-Intrusive Load Monitoring
• Bootstrap Your Flow