Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
arXiv:2102.11086
Abstract
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder, but we show how to drastically reduce this additional cost with couplings in the latent space. When parallel architectures can be exploited, our coders can achieve better rates than bits-back with little additional cost. We demonstrate improved lossless compression rates in a variety of settings, especially in out-of-distribution or sequential data compression.
References in corpus (22)
- Learning Transferable Visual Models From Natural Language Supervision
- Categorical Reparameterization with Gumbel-Softmax
- Neural Discrete Representation Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Variational image compression with a scale hyperprior
- NICE: Non-linear Independent Components Estimation
- End-to-end Optimized Image Compression
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Density estimation using Real NVP
- A Recurrent Latent Variable Model for Sequential Data
- Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
- Joint Autoregressive and Hierarchical Priors for Learned Image Compression
- Asymmetric numeral systems
- High-Dimensional Probability Estimation with Deep Density Models
- Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
- Improving Inference for Neural Image Compression
- Reinterpreting Importance-Weighted Autoencoders
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- Importance Weighting and Variational Inference
- IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
- Computationally Efficient Neural Image Compression
- A tutorial on the range variant of asymmetric numeral systems