IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
arXiv:2006.12459
Abstract
In this paper we analyse and improve integer discrete flows for lossless compression. Integer discrete flows are a recently proposed class of models that learn invertible transformations for integer-valued random variables. Their discrete nature makes them particularly suitable for lossless compression with entropy coding schemes. We start by investigating a recent theoretical claim that states that invertible flows for discrete random variables are less flexible than their continuous counterparts. We demonstrate with a proof that this claim does not hold for integer discrete flows due to the embedding of data with finite support into the countably infinite integer lattice. Furthermore, we zoom in on the effect of gradient bias due to the straight-through estimator in integer discrete flows, and demonstrate that its influence is highly dependent on architecture choices and less prominent than previously thought. Finally, we show how different architecture modifications improve the performance of this model class for lossless compression, and that they also enable more efficient compression: a model with half the number of flow layers performs on par with or better than the original integer discrete flow model.
arXiv commentAccepted as a conference paper at the Ninth International Conference on Learning Representations (ICLR) 2021
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Searching for Activation Functions
- Density estimation using Real NVP
- A note on the evaluation of generative models
- Masked Autoregressive Flow for Density Estimation
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- ReZero is All You Need: Fast Convergence at Large Depth
- Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Generative Image Modeling Using Spatial LSTMs
- RNADE: The real-valued neural autoregressive density-estimator
- A Deep and Tractable Density Estimator
- Asymmetric numeral systems
- Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
- Universally Quantized Neural Compression
- Practical Lossless Compression with Latent Variables using Bits Back Coding
- Emerging Convolutions for Generative Normalizing Flows
- Discrete Flows: Invertible Generative Models of Discrete Data
- Compression with Flows via Local Bits-Back Coding
- Lossless Image Compression through Super-Resolution
- FloWaveNet : A Generative Flow for Raw Audio
- Neural Spline Flows
- Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
- A tutorial on the range variant of asymmetric numeral systems
Cited by in corpus (15)
- Normalizing Flows for Probabilistic Modeling and Inference
- Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
- Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
- Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
- Ephemeral Learning -- Augmenting Triggers with Online-Trained Normalizing Flows
- A RAD approach to deep mixture models
- Unified learning-based lossy and lossless JPEG recompression
- Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization
- L3C-Stereo: Lossless Compression for Stereo Images
- On the Out-of-distribution Generalization of Probabilistic Image Modelling
- Learning Non-linear Wavelet Transformation via Normalizing Flow
- Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
- General Invertible Transformations for Flow-based Generative Modeling
- iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression
- Lossless Compression with Latent Variable Models