Variational Bayesian Quantization
arXiv:2002.08158
Abstract
We propose a novel algorithm for quantizing continuous latent representations in trained models. Our approach applies to deep probabilistic models, such as variational autoencoders (VAEs), and enables both data and model compression. Unlike current end-to-end neural compression methods that cater the model to a fixed quantization scheme, our algorithm separates model design and training from quantization. Consequently, our algorithm enables "plug-and-play" compression with variable rate-distortion trade-off, using a single trained model. Our algorithm can be seen as a novel extension of arithmetic coding to the continuous domain, and uses adaptive quantization accuracy based on estimates of posterior uncertainty. Our experimental results demonstrate the importance of taking into account posterior uncertainties, and show that image compression with the proposed algorithm outperforms JPEG over a wide range of bit rates using only a single standard VAE. Further experiments on Bayesian neural word embeddings demonstrate the versatility of the proposed method.
arXiv comment9 pages + detailed supplement with additional full resolution reconstructed images; ICML 2020 final camera-ready version, title changed to "Variational Bayesian Quantization" following reviewer feedback
References in corpus (13)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- Auto-Encoding Variational Bayes
- Variational image compression with a scale hyperprior
- End-to-end Optimized Image Compression
- Weight Uncertainty in Neural Networks
- Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
- Video Compression With Rate-Distortion Autoencoders
- Dynamic Word Embeddings
- Towards Conceptual Compression
- Improving Inference for Neural Image Compression
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- Bayesian Neural Word Embedding