Normalizing Flows for Probabilistic Modeling and Inference

Abstract

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

arXiv commentReview article, 64 pages, 9 figures. Published in the Journal of Machine Learning Research (see https://jmlr.org/papers/v22/19-1028.html)

References in corpus (51)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Spectral Normalization for Generative Adversarial Networks
• Conditional Image Generation with PixelCNN Decoders
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Normalizing Flows: An Introduction and Review of Current Methods
• The frontier of simulation-based inference
• MADE: Masked Autoencoder for Distribution Estimation
• Guided Image Generation with Conditional Invertible Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Learning in Implicit Generative Models
• Residual Flows for Invertible Generative Modeling
• Constraining Effective Field Theories with Machine Learning
• Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
• Iterative Gaussianization: from ICA to Random Rotations
• Generative Image Modeling Using Spatial LSTMs
• RNADE: The real-valued neural autoregressive density-estimator
• A Deep and Tractable Density Estimator
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Improving Variational Auto-Encoders using Householder Flow
• A-NICE-MC: Adversarial Training for MCMC
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
• High-Dimensional Probability Estimation with Deep Density Models
• Learning Likelihoods with Conditional Normalizing Flows
• Latent Normalizing Flows for Discrete Sequences
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• Emerging Convolutions for Generative Normalizing Flows
• Comparison of Maximum Likelihood and GAN-based training of Real NVPs
• Equivariant Hamiltonian Flows
• Discrete Flows: Invertible Generative Models of Discrete Data
• Sum-of-Squares Polynomial Flow
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets
• Hybrid Models with Deep and Invertible Features
• Graph Residual Flow for Molecular Graph Generation
• Cubic-Spline Flows
• Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
• A RAD approach to deep mixture models
• Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• Neural Networks with Cheap Differential Operators
• Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling
• Continuous Graph Flow

Cited by in corpus (117)

• Normalizing Flows: An Introduction and Review of Current Methods
• Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
• Nested Sampling with Normalising Flows for Gravitational-Wave Inference
• Mapping the Ethics of Generative AI: A Comprehensive Scoping Review
• Likelihood-free inference with neural compression of DES SV weak lensing map statistics
• Estimation of Thermodynamic Observables in Lattice Field Theories with Deep Generative Models
• A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
• Understanding Event-Generation Networks via Uncertainties
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Normalizing Flows on Tori and Spheres
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Generative Networks for Precision Enthusiasts
• Rare and Different: Anomaly Scores from a combination of likelihood and out-of-distribution models to detect new physics at the LHC
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
• Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Ensuring thermodynamic consistency with invertible coarse-graining
• Inference of cosmic-ray source properties by conditional invertible neural networks
• Riemannian Continuous Normalizing Flows
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
• On the representation and learning of monotone triangular transport maps
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• Probabilistic Autoencoder
• Bayesian Stokes inversion with Normalizing flows
• CaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• A Flow-Based Neural Network for Time Domain Speech Enhancement
• Neural Canonical Transformation with Symplectic Flows
• Deep composition of tensor-trains using squared inverse Rosenblatt transports
• Predictive Coding: a Theoretical and Experimental Review
• Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model
• The LSST-DESC 3x2pt Tomography Optimization Challenge
• Perfect density models cannot guarantee anomaly detection
• Conditional Set Generation with Transformers
• Learning the temporal evolution of multivariate densities via normalizing flows
• Benchmarking Simulation-Based Inference
• Universal Approximation Property of Neural Ordinary Differential Equations
• Multimap targeted free energy estimation
• Wavelet Flow: Fast Training of High Resolution Normalizing Flows
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Charting Galactic Accelerations: When and How to Extract a Unique Potential from the Distribution Function
• STEER: Simple Temporal Regularization For Neural ODEs
• Stochastic Normalizing Flows
• Particle-based Energetic Variational Inference
• Introduction to Normalizing Flows for Lattice Field Theory
• Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems
• Neural canonical transformations for vibrational spectra of molecules
• Muesli: Combining Improvements in Policy Optimization
• Graphical Normalizing Flows
• UnICORNN: A recurrent model for learning very long time dependencies
• Modeling Extremal Streamflow using Deep Learning Approximations and a Flexible Spatial Process
• Deep Normalization for Speaker Vectors
• Learning Gradient Fields for Shape Generation
• Disentangling Geometric Deformation Spaces in Generative Latent Shape Models
• Transforming Gaussian Processes With Normalizing Flows
• Normalizing flows for random fields in cosmology
• Conditional Image Generation with Score-Based Diffusion Models
• PROUD: PaRetO-gUided Diffusion Model for Multi-objective Generation
• Sequential Neural Posterior and Likelihood Approximation
• Exact nonlinear state estimation
• Variational inference based on a subclass of closed skew normals
• Adaptive Monte Carlo augmented with normalizing flows
• Normalizing Flows Across Dimensions
• Deep Conditional Transformation Models
• Generative Machine Learning for Multivariate Equity Returns
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• Addressing Correlated Latent Exogenous Variables in Debiased Recommender Systems
• Self-Tuning Hamiltonian Monte Carlo for Accelerated Sampling
• Annealed Flow Transport Monte Carlo
• Semi-supervised Learning with Missing Values Imputation
• Deep Speaker Vector Normalization with Maximum Gaussianality Training
• Causal Inference with Deep Causal Graphs
• MC-LSTM: Mass-Conserving LSTM
• Sum-Product-Transform Networks: Exploiting Symmetries using Invertible Transformations
• An In-depth Summary of Recent Artificial Intelligence Applications in Drug Design
• Principled Interpolation in Normalizing Flows
• Nonreversible MCMC from conditional invertible transforms: a complete recipe with convergence guarantees
• Stein Variational Gaussian Processes
• On the Discrepancy between Density Estimation and Sequence Generation
• The Convolution Exponential and Generalized Sylvester Flows
• Variational Inference with Holder Bounds
• NF-iSAM: Incremental Smoothing and Mapping via Normalizing Flows
• Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems
• ASR-Free Pronunciation Assessment
• Uncertainty quantification in medical image segmentation with normalizing flows
• Learning Dynamics Models with Stable Invariant Sets
• NEO: Non Equilibrium Sampling on the Orbit of a Deterministic Transform
• Are Bayesian neural networks intrinsically good at out-of-distribution detection?
• Measure Transport with Kernel Stein Discrepancy
• Harmonization with Flow-based Causal Inference
• Rectangular Flows for Manifold Learning
• Deep generative LDA
• Advances in Black-Box VI: Normalizing Flows, Importance Weighting, and Optimization
• Robust model training and generalisation with Studentising flows
• Learning Bijective Feature Maps for Linear ICA
• The Phantom Steering Effect in Q&A Websites
• Density Deconvolution with Normalizing Flows
• Causal Autoregressive Flows
• Scaling Bayesian inference of mixed multinomial logit models to very large datasets
• Relaxed-Responsibility Hierarchical Discrete VAEs
• Neural Discriminant Analysis for Deep Speaker Embedding
• Semi-Supervised Few-Shot Classification with Deep Invertible Hybrid Models
• Estimation of Bivariate Structural Causal Models by Variational Gaussian Process Regression Under Likelihoods Parametrised by Normalising Flows
• Deep generative factorization for speech signal
• Continuous normalizing flows on manifolds
• Uncertainty estimation under model misspecification in neural network regression
• Optimal Design of Experiments for Simulation-Based Inference of Mechanistic Acyclic Biological Networks
• Can We Trust Deep Speech Prior?
• Few-shot time series segmentation using prototype-defined infinite hidden Markov models
• Improving the expressiveness of neural vocoding with non-affine Normalizing Flows
• Abelian Neural Networks
• Latent Transformations for Discrete-Data Normalising Flows
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Langevin Cooling for Domain Translation
• Gradient Boosted Normalizing Flows