Discrete Flows: Invertible Generative Models of Discrete Data

Abstract

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We consider two flow architectures: discrete autoregressive flows that enable bidirectionality, allowing, for example, tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows that enable efficient non-autoregressive generation as in RealNVP. Empirically, we find that discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models; and bipartite flows can obtain competitive performance with autoregressive baselines on character-level language modeling for Penn Tree Bank and text8.

References in corpus (10)

• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Unrolled Generative Adversarial Networks
• Learning to Execute
• Fast Decoding in Sequence Models using Discrete Latent Variables
• An Analysis of Neural Language Modeling at Multiple Scales
• High-Dimensional Probability Estimation with Deep Density Models
• Latent Normalizing Flows for Discrete Sequences

Cited by in corpus (25)

• Normalizing Flows: An Introduction and Review of Current Methods
• Normalizing Flows for Probabilistic Modeling and Inference
• Equivariant flow-based sampling for lattice gauge theory
• Asymptotically unbiased estimation of physical observables with neural samplers
• i-flow: High-dimensional Integration and Sampling with Normalizing Flows
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows
• WaveFlow: A Compact Flow-based Model for Raw Audio
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• A RAD approach to deep mixture models
• Learning Discrete Distributions by Dequantization
• Categorical Normalizing Flows via Continuous Transformations
• Integer Discrete Flows and Lossless Compression
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• Weakly Supervised Label Learning Flows
• Flow Contrastive Estimation of Energy-Based Models
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• Woodbury Transformations for Deep Generative Flows
• Relaxed-Responsibility Hierarchical Discrete VAEs
• Reliable Categorical Variational Inference with Mixture of Discrete Normalizing Flows
• Partition and Code: learning how to compress graphs
• Boosting Summarization with Normalizing Flows and Aggressive Training