HEMP: High-order Entropy Minimization for neural network comPression

Abstract

We formulate the entropy of a quantized artificial neural network as a differentiable function that can be plugged as a regularization term into the cost function minimized by gradient descent. Our formulation scales efficiently beyond the first order and is agnostic of the quantization scheme. The network can then be trained to minimize the entropy of the quantized parameters, so that they can be optimally compressed via entropy coding. We experiment with our entropy formulation at quantizing and compressing well-known network architectures over multiple datasets. Our approach compares favorably over similar methods, enjoying the benefits of higher order entropy estimate, showing flexibility towards non-uniform quantization (we use Lloyd-max quantization), scalability towards any entropy order to be minimized and efficiency in terms of compression. We show that HEMP is able to work in synergy with other approaches aiming at pruning or quantizing the model itself, delivering significant benefits in terms of storage size compressibility without harming the model's performance.

References in corpus (15)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Distributed Federated Learning for Ultra-Reliable Low-Latency Vehicular Communications
• Towards Accurate Binary Convolutional Neural Network
• Variational Dropout Sparsifies Deep Neural Networks
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• Deep Neural Network Compression with Single and Multiple Level Quantization
• LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
• On the role of synaptic stochasticity in training low-precision neural networks
• Learning Discrete Weights Using the Local Reparameterization Trick
• Scalable Model Compression by Entropy Penalized Reparameterization

Cited by in corpus (2)

• Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?
• Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs