Learning Discrete Distributions by Dequantization

Abstract

Media is generally stored digitally and is therefore discrete. Many successful deep distribution models in deep learning learn a density, i.e., the distribution of a continuous random variable. Naïve optimization on discrete data leads to arbitrarily high likelihoods, and instead, it has become standard practice to add noise to datapoints. In this paper, we present a general framework for dequantization that captures existing methods as a special case. We derive two new dequantization objectives: importance-weighted (iw) dequantization and Rényi dequantization. In addition, we introduce autoregressive dequantization (ARD) for more flexible dequantization distributions. Empirically we find that iw and Rényi dequantization considerably improve performance for uniform dequantization distributions. ARD achieves a negative log-likelihood of 3.06 bits per dimension on CIFAR10, which to the best of our knowledge is state-of-the-art among distribution models that do not require autoregressive inverses for sampling.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Pixel Recurrent Neural Networks
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• High-Dimensional Probability Estimation with Deep Density Models
• Learning Likelihoods with Conditional Normalizing Flows
• Importance Weighting and Variational Inference
• Emerging Convolutions for Generative Normalizing Flows
• Discrete Flows: Invertible Generative Models of Discrete Data
• MintNet: Building Invertible Neural Networks with Masked Convolutions

Cited by in corpus (9)

• Normalizing Flows: An Introduction and Review of Current Methods
• Wavelet Flow: Fast Training of High Resolution Normalizing Flows
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Categorical Normalizing Flows via Continuous Transformations
• Invertible DenseNets with Concatenated LipSwish
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow
• Manifold Density Estimation via Generalized Dequantization
• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression