Practical Lossless Compression with Latent Variables using Bits Back Coding

Abstract

Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back .

References in corpus (8)

• Auto-Encoding Variational Bayes
• Binarized Neural Networks
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• Composing graphical models with neural networks for structured representations and fast inference
• PixelVAE: A Latent Variable Model for Natural Images
• Asymmetric numeral systems
• Soft Weight-Sharing for Neural Network Compression
• Interleaved entropy coders

Cited by in corpus (26)

• Variational Diffusion Models
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Improving Inference for Neural Image Compression
• Maximum Likelihood Training of Score-Based Diffusion Models
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Ephemeral Learning -- Augmenting Triggers with Online-Trained Normalizing Flows
• A tutorial on the range variant of asymmetric numeral systems
• Integer Discrete Flows and Lossless Compression
• Hierarchical VAEs Know What They Don't Know
• Improving VAEs' Robustness to Adversarial Attack
• Towards Empirical Sandwich Bounds on the Rate-Distortion Function
• Compressing Multisets with Large Alphabets using Bits-Back Coding
• Lossless compression with state space models using bits back coding
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks
• Relaxed-Responsibility Hierarchical Discrete VAEs
• Partition and Code: learning how to compress graphs
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Autoregressive Diffusion Models
• Neural Communication Systems with Bandwidth-limited Channel
• Variational Bayesian Quantization
• Lossless Compression with Probabilistic Circuits
• Algorithms for the Communication of Samples
• Entropic Issues in Likelihood-Based OOD Detection
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression