Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows

Abstract

We show that normalising flows become pathological when used to model targets whose supports have complicated topologies. In this scenario, we prove that a flow must become arbitrarily numerically noninvertible in order to approximate the target closely. This result has implications for all flow-based models, and especially Residual Flows (ResFlows), which explicitly control the Lipschitz constant of the bijection used. To address this, we propose Continuously Indexed Flows (CIFs), which replace the single bijection used by normalising flows with a continuously indexed family of bijections, and which can intuitively "clean up" mass that would otherwise be misplaced by a single bijection. We show theoretically that CIFs are not subject to the same topological limitations as normalising flows, and obtain better empirical performance on a variety of models and benchmarks.

Minor revision

References in corpus (14)

• Auto-Encoding Variational Bayes
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Spectral Normalization for Generative Adversarial Networks
• Semi-Supervised Learning with Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• MADE: Masked Autoencoder for Distribution Estimation
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Residual Flows for Invertible Generative Modeling
• On Russian Roulette Estimates for Bayesian Inference with Doubly-Intractable Likelihoods
• Sum-of-Squares Polynomial Flow
• A RAD approach to deep mixture models
• Efficient Optimization of Loops and Limits with Randomized Telescoping Sums

Cited by in corpus (12)

• Normalizing Flows for Probabilistic Modeling and Inference
• CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Deep composition of tensor-trains using squared inverse Rosenblatt transports
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Stochastic Normalizing Flows
• The Power Spherical distribution
• Improved Autoregressive Modeling with Distribution Smoothing
• Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows
• Latent Transformations for Discrete-Data Normalising Flows
• Entropy-based adaptive Hamiltonian Monte Carlo
• Gradient Boosted Normalizing Flows