Categorical Normalizing Flows via Continuous Transformations
arXiv:2006.09790
Abstract
Despite their popularity, to date, the application of normalizing flows on categorical data stays limited. The current practice of using dequantization to map discrete data to a continuous space is inapplicable as categorical data has no intrinsic order. Instead, categorical data have complex and latent relations that must be inferred, like the synonymy between words. In this paper, we investigate \emph{Categorical Normalizing Flows}, that is normalizing flows for categorical data. By casting the encoding of categorical data in continuous space as a variational inference problem, we jointly optimize the continuous representation and the model likelihood. Using a factorized decoder, we introduce an inductive bias to model any interactions in the normalizing flow. As a consequence, we do not only simplify the optimization compared to having a joint decoder, but also make it possible to scale up to a large number of categories that is currently impossible with discrete normalizing flows. Based on Categorical Normalizing Flows, we propose GraphCNF a permutation-invariant generative model on graphs. GraphCNF implements a three step approach modeling the nodes, edges and adjacency matrix stepwise to increase efficiency. On molecule generation, GraphCNF outperforms both one-shot and autoregressive flow-based state-of-the-art.
arXiv commentSubmitted to: International Conference on Learning Representations (ICLR), 2021
References in corpus (19)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Gaussian Error Linear Units (GELUs)
- Graph Neural Networks: A Review of Methods and Applications
- Variational Inference with Normalizing Flows
- Junction Tree Variational Autoencoder for Molecular Graph Generation
- GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
- Learning Deep Generative Models of Graphs
- Constrained Graph Variational Autoencoders for Molecule Design
- GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- Efficient Graph Generation with Graph Recurrent Attention Networks
- RNADE: The real-valued neural autoregressive density-estimator
- Improving Variational Auto-Encoders using Householder Flow
- Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders
- Latent Normalizing Flows for Discrete Sequences
- Graph Normalizing Flows
- Discrete Flows: Invertible Generative Models of Discrete Data
- Learning Discrete Distributions by Dequantization