Scalable Model Compression by Entropy Penalized Reparameterization

Abstract

We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training. Classification accuracy and model compressibility is maximized jointly, with the bitrate--accuracy trade-off specified by a hyperparameter. We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.

arXiv commentPublished in ICLR 2020

References in corpus (9)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Variational image compression with a scale hyperprior
• End-to-end Optimized Image Compression
• Ternary Weight Networks
• Pruning Filters for Efficient ConvNets
• Compressing Neural Networks with the Hashing Trick
• Bayesian Compression for Deep Learning
• Lossy Image Compression with Compressive Autoencoders
• DeepCABAC: Context-adaptive binary arithmetic coding for deep neural network compression

Cited by in corpus (2)

• HEMP: High-order Entropy Minimization for neural network comPression
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression