DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

Abstract

The field of video compression has developed some of the most sophisticated and efficient compression algorithms known in the literature, enabling very high compressibility for little loss of information. Whilst some of these techniques are domain specific, many of their underlying principles are universal in that they can be adapted and applied for compressing different types of data. In this work we present DeepCABAC, a compression algorithm for deep neural networks that is based on one of the state-of-the-art video coding techniques. Concretely, it applies a Context-based Adaptive Binary Arithmetic Coder (CABAC) to the network's parameters, which was originally designed for the H.264/AVC video coding standard and became the state-of-the-art for lossless compression. Moreover, DeepCABAC employs a novel quantization scheme that minimizes the rate-distortion function while simultaneously taking the impact of quantization onto the accuracy of the network into account. Experimental results show that DeepCABAC consistently attains higher compression rates than previously proposed coding techniques for neural network compression. For instance, it is able to compress the VGG16 ImageNet model by x63.6 with no loss of accuracy, thus being able to represent the entire network with merely 8.7MB. The source code for encoding and decoding can be found at https://github.com/fraunhoferhhi/DeepCABAC.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Deep Learning with Limited Numerical Precision
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• cuDNN: Efficient Primitives for Deep Learning
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Trained Ternary Quantization
• Scalable Methods for 8-bit Training of Neural Networks
• Dynamic Network Surgery for Efficient DNNs
• Critical Learning Periods in Deep Neural Networks
• Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
• Improved Bayesian Compression

Cited by in corpus (4)

• ECQ${}^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs
• Trends and Advancements in Deep Neural Network Communication
• Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
• Prune Your Model Before Distill It