Lossless Compression with Latent Variable Models

Abstract

We develop a simple and elegant method for lossless compression using latent variable models, which we call 'bits back with asymmetric numeral systems' (BB-ANS). The method involves interleaving encode and decode steps, and achieves an optimal rate when compressing batches of data. We demonstrate it firstly on the MNIST test set, showing that state-of-the-art lossless compression is possible using a small variational autoencoder (VAE) model. We then make use of a novel empirical insight, that fully convolutional generative models, trained on small images, are able to generalize to images of arbitrary size, and extend BB-ANS to hierarchical latent variable models, enabling state-of-the-art lossless compression of full-size colour images from the ImageNet dataset. We describe 'Craystack', a modular software framework which we have developed for rapid prototyping of compression using deep generative models.

References in corpus (14)

• Auto-Encoding Variational Bayes
• Language Models are Few-Shot Learners
• Pixel Recurrent Neural Networks
• Binarized Neural Networks
• NVAE: A Deep Hierarchical Variational Autoencoder
• MADE: Masked Autoencoder for Distribution Estimation
• Asymmetric numeral systems
• Soft Weight-Sharing for Neural Network Compression
• Compression with Flows via Local Bits-Back Coding
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Interleaved entropy coders
• Lightweight compression with encryption based on Asymmetric Numeral Systems
• A tutorial on the range variant of asymmetric numeral systems
• Lossless compression with state space models using bits back coding