Latent Normalizing Flows for Discrete Sequences

Abstract

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.

Cited by in corpus (28)

• Normalizing Flows: An Introduction and Review of Current Methods
• Normalizing Flows for Probabilistic Modeling and Inference
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• Discrete Flows: Invertible Generative Models of Discrete Data
• Copula Flows for Synthetic Data Generation
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Neural Spline Flows
• Intensity-Free Learning of Temporal Point Processes
• Categorical Normalizing Flows via Continuous Transformations
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings
• Structured Output Learning with Conditional Generative Flows
• Do sequence-to-sequence VAEs learn global features of sentences?
• MaCow: Masked Convolutional Generative Flow
• APo-VAE: Text Generation in Hyperbolic Space
• Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models
• Variational Dynamic Mixtures
• PassFlow: Guessing Passwords with Generative Flows
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
• Towards Recurrent Autoregressive Flow Models
• Adversarial Stein Training for Graph Energy Models
• Prior Flow Variational Autoencoder: A density estimation model for Non-Intrusive Load Monitoring
• Cross-Domain Latent Modulation for Variational Transfer Learning
• Uncertainty quantification for ptychography using normalizing flows