Integer Discrete Flows and Lossless Compression
arXiv:1905.07376
Abstract
Lossless compression methods shorten the expected representation size of data without loss of information, using a statistical model. Flow-based models are attractive in this setting because they admit exact likelihood optimization, which is equivalent to minimizing the expected number of bits per message. However, conventional flows assume continuous data, which may lead to reconstruction errors when quantized for compression. For that reason, we introduce a flow-based generative model for ordinal discrete data called Integer Discrete Flow (IDF): a bijective integer map that can learn rich transformations on high-dimensional data. As building blocks for IDFs, we introduce a flexible transformation layer called integer discrete coupling. Our experiments show that IDFs are competitive with other flow-based generative models. Furthermore, we demonstrate that IDF based compression achieves state-of-the-art lossless compression rates on CIFAR10, ImageNet32, and ImageNet64. To the best of our knowledge, this is the first lossless compression method that uses invertible neural networks.
Accepted as a conference paper at Neural Information Processing Systems (NeurIPS) 2019
References in corpus (12)
- Auto-Encoding Variational Bayes
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding
- Asymmetric numeral systems
- High-Dimensional Probability Estimation with Deep Density Models
- Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- Emerging Convolutions for Generative Normalizing Flows
- Discrete Flows: Invertible Generative Models of Discrete Data