Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow

Abstract

Flow models have recently made great progress at modeling ordinal discrete data such as images and audio. Due to the continuous nature of flow models, dequantization is typically applied when using them for such discrete data, resulting in lower bound estimates of the likelihood. In this paper, we introduce subset flows, a class of flows that can tractably transform finite volumes and thus allow exact computation of likelihoods for discrete data. Based on subset flows, we identify ordinal discrete autoregressive models, including WaveNets, PixelCNNs and Transformers, as single-layer flows. We use the flow formulation to compare models trained and evaluated with either the exact likelihood or its dequantization lower bound. Finally, we study multilayer flows composed of PixelCNNs and non-autoregressive coupling layers and demonstrate state-of-the-art results on CIFAR-10 for flow models trained with dequantization.

References in corpus (15)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Conditional Image Generation with PixelCNN Decoders
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Pixel Recurrent Neural Networks
• Variational Inference with Normalizing Flows
• Generating Long Sequences with Sparse Transformers
• MADE: Masked Autoencoder for Distribution Estimation
• Image Transformer
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• RNADE: The real-valued neural autoregressive density-estimator
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Discrete Flows: Invertible Generative Models of Discrete Data
• Sum-of-Squares Polynomial Flow
• Learning Discrete Distributions by Dequantization