A RAD approach to deep mixture models

Abstract

Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for modeling and representing discrete structures in data distributions, for example class membership or discrete symmetries. To address this difficulty, we present a normalizing flow architecture which relies on domain partitioning using locally invertible functions, and possesses both real and discrete valued latent variables. This Real and Discrete (RAD) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution.

arXiv comment18.5 pages of main content, 3 pages of appendices

References in corpus (19)

• Auto-Encoding Variational Bayes
• The NumPy array: a structure for efficient numerical computation
• Neural Discrete Representation Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• NICE: Non-linear Independent Components Estimation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• On the Number of Linear Regions of Deep Neural Networks
• Neural Variational Inference and Learning in Belief Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• On GANs and GMMs
• High-Dimensional Probability Estimation with Deep Density Models
• Tackling Over-pruning in Variational Autoencoders
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• Discrete Flows: Invertible Generative Models of Discrete Data
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Deep Mixtures of Factor Analysers
• Discriminator Rejection Sampling
• Inference Trees: Adaptive Inference with Exploration

Cited by in corpus (16)

• Normalizing Flows: An Introduction and Review of Current Methods
• Normalizing Flows for Probabilistic Modeling and Inference
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Stabilizing Invertible Neural Networks Using Mixture Models
• Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows
• Stochastic Normalizing Flows
• The Power Spherical distribution
• Max-Affine Spline Insights into Deep Generative Networks
• Featurized Density Ratio Estimation
• Variational Mixture of Normalizing Flows
• Decoupling Global and Local Representations via Invertible Generative Flows
• Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows
• Latent Transformations for Discrete-Data Normalising Flows
• Diffusion Normalizing Flow