Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling

Abstract

Normalizing flows have shown great success as general-purpose density estimators. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used to induce desirable properties such as multimodality, hierarchical coupling and continuity. Furthermore, we show that EMFs enable a high performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms state-of-the-art methods in common structured inference problems.

References in corpus (19)

• Auto-Encoding Variational Bayes
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Normalizing Flows: An Introduction and Review of Current Methods
• Edward: A library for probabilistic modeling, inference, and criticism
• Flow-based generative models for Markov chain Monte Carlo in lattice field theory
• Equivariant flow-based sampling for lattice gauge theory
• Implicit Reparameterization Gradients
• Estimation of Thermodynamic Observables in Lattice Field Theories with Deep Generative Models
• Deep Probabilistic Programming
• Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows
• Equivariant Hamiltonian Flows
• Differentiable Strong Lensing: Uniting Gravity and Neural Nets through Differentiable Probabilistic Programming
• Neural Canonical Transformation with Symplectic Flows
• Introduction to Normalizing Flows for Lattice Field Theory
• Joint Distributions for TensorFlow Probability
• Targeted Likelihood-Free Inference of Dark Matter Substructure in Strongly-Lensed Galaxies
• Dark Matter Subhalos, Strong Lensing and Machine Learning
• Normalizing Flows as a Novel PDF Turbulence Model

Cited by in corpus (1)

• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling