Residual Flows for Invertible Generative Modeling

Abstract

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforcing invertibility. However, prior work trained invertible residual networks for density estimation by relying on biased log-density estimates whose bias increased with the network's expressiveness. We give a tractable unbiased estimate of the log density using a "Russian roulette" estimator, and reduce the memory required during training by using an alternative infinite series for the gradient. Furthermore, we improve invertible residual blocks by proposing the use of activation functions that avoid derivative saturation and generalizing the Lipschitz condition to induced mixed norms. The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.

NeurIPS 2019

Cited by in corpus (78)

• Score-Based Generative Modeling through Stochastic Differential Equations
• Normalizing Flows: An Introduction and Review of Current Methods
• NVAE: A Deep Hierarchical Variational Autoencoder
• Normalizing Flows for Probabilistic Modeling and Inference
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities
• Riemannian Continuous Normalizing Flows
• Stochastic Normalizing Flows for Inverse Problems: a Markov Chains Viewpoint
• Universal Approximation with Deep Narrow Networks
• The Lipschitz Constant of Self-Attention
• Deep composition of tensor-trains using squared inverse Rosenblatt transports
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
• Variational Autoencoders with Normalizing Flow Decoders
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• iUNets: Fully invertible U-Nets with Learnable Up- and Downsampling
• On Training Implicit Models
• Tail of Distribution GAN (TailGAN): Generative-Adversarial-Network-Based Boundary Formation
• Boundary of Distribution Support Generator (BDSG): Sample Generation on the Boundary
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• Densely connected normalizing flows
• Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference
• Invertible DenseNets with Concatenated LipSwish
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• Large-time asymptotics in deep learning
• Invertible Generative Modeling using Linear Rational Splines
• Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
• Optimizing Functionals on the Space of Probabilities with Input Convex Neural Networks
• Randomized Automatic Differentiation
• Hybrid Models for Open Set Recognition
• Super-resolution Variational Auto-Encoders
• Path Integral Sampler: a stochastic control approach for sampling
• Normalizing Flows Across Dimensions
• Implicit Normalizing Flows
• Hybrid Generative-Contrastive Representation Learning
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• Sparse Flows: Pruning Continuous-depth Models
• On the expressivity of bi-Lipschitz normalizing flows
• Principled Interpolation in Normalizing Flows
• Task-agnostic Continual Learning with Hybrid Probabilistic Models
• The Convolution Exponential and Generalized Sylvester Flows
• Invertible DenseNets
• Automatic variational inference with cascading flows
• Jacobian Determinant of Normalizing Flows
• Stability and Generalization of Bilevel Programming in Hyperparameter Optimization
• Distilling the Knowledge from Conditional Normalizing Flows
• Rectangular Flows for Manifold Learning
• Efficient and Accurate Gradients for Neural SDEs
• Bias-Free Scalable Gaussian Processes via Randomized Truncations
• Woodbury Transformations for Deep Generative Flows
• Quadruply Stochastic Gaussian Processes
• Two equalities expressing the determinant of a matrix in terms of expectations over matrix-vector products
• Training Invertible Linear Layers through Rank-One Perturbations
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy
• General Invertible Transformations for Flow-based Generative Modeling
• Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization
• PassFlow: Guessing Passwords with Generative Flows
• Variational Determinant Estimation with Spherical Normalizing Flows
• Normalizing Flows with Multi-Scale Autoregressive Priors
• Decoupling Global and Local Representations via Invertible Generative Flows
• Copula-Based Normalizing Flows
• Independent mechanism analysis, a new concept?
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Entropy-based adaptive Hamiltonian Monte Carlo
• Generalization of the Change of Variables Formula with Applications to Residual Flows
• Viscos Flows: Variational Schur Conditional Sampling With Normalizing Flows
• Continuous Latent Process Flows
• Gradient Boosted Normalizing Flows
• On the Generative Utility of Cyclic Conditionals
• Bootstrap Your Flow
• Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning
• Diffusion Normalizing Flow
• Self Normalizing Flows