Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence

Abstract

In collaborative intelligence applications, part of a deep neural network (DNN) is deployed on a lightweight device such as a mobile phone or edge device, and the remaining portion of the DNN is processed where more computing resources are available, such as in the cloud. This paper presents a novel lightweight compression technique designed specifically to quantize and compress the features output by the intermediate layer of a split DNN, without requiring any retraining of the network weights. Mathematical models for estimating the clipping and quantization error of ReLU and leaky-ReLU activations at this intermediate layer are developed and used to compute optimal clipping ranges for coarse quantization. We also present a modified entropy-constrained design algorithm for quantizing clipped activations. When applied to popular object-detection and classification DNNs, we were able to compress the 32-bit floating point intermediate activations down to 0.6 to 0.8 bits, while keeping the loss in accuracy to less than 1%. When compared to HEVC, we found that the lightweight codec consistently provided better inference accuracy, by up to 1.3%. The performance and simplicity of this lightweight compression technique makes it an attractive option for coding an intermediate layer of a split neural network for edge/cloud applications.

arXiv commentAccepted for publication in IEEE Open Journal of Circuits and Systems

References in corpus (8)

• YOLOv3: An Incremental Improvement
• Binarized Neural Networks
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Scalable Methods for 8-bit Training of Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• Lightweight compression of neural network feature tensors for collaborative intelligence

Cited by in corpus (5)

• Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges
• Flexible Variable-Rate Image Feature Compression for Edge-Cloud Systems
• Efficient Feature Compression for Machines with Global Statistics Preservation
• Inter-Feature-Map Differential Coding of Surveillance Video
• DFTS2: Simulating Deep Feature Transmission Over Packet Loss Channels