Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

Abstract

The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generalizes BB-ANS and achieves strictly better compression rates for hierarchical latent variable models with Markov chain structure. Through experiments we verify that Bit-Swap results in lossless compression rates that are empirically superior to existing techniques. Our implementation is available at https://github.com/fhkingma/bitswap.

Accepted to ICML 2019

References in corpus (11)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Neural Machine Translation in Linear Time
• Variational Lossy Autoencoder
• Image Transformer
• Asymmetric numeral systems
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Practical Lossless Compression with Latent Variables using Bits Back Coding

Cited by in corpus (16)

• Variational Diffusion Models
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Improving Inference for Neural Image Compression
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Unified learning-based lossy and lossless JPEG recompression
• Integer Discrete Flows and Lossless Compression
• Towards Empirical Sandwich Bounds on the Rate-Distortion Function
• Lossless compression with state space models using bits back coding
• Compressing Multisets with Large Alphabets using Bits-Back Coding
• On the Out-of-distribution Generalization of Probabilistic Image Modelling
• Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks
• Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
• Partition and Code: learning how to compress graphs
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression
• Lossless Compression with Probabilistic Circuits
• Bayesian Attention Networks for Data Compression