Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

Abstract

This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters. We explore the previously overlooked opportunity of cross-layer architecture-agnostic representation sharing for DNN parameters. To do this, we decouple feedforward parameters from DNN architectures and leverage additive quantization, an extreme lossy compression method invented for image descriptors, to compactly represent the parameters. The representations are then finetuned on task objectives to improve task accuracy. We conduct extensive experiments on MobileNet-v2, VGG-11, ResNet-50, Feature Pyramid Networks, and pruned DNNs trained for classification, detection, and segmentation tasks. The conceptually simple scheme consistently outperforms iterative unstructured pruning. Applied to ResNet-50 with 76.1% top-1 accuracy on the ILSVRC12 classification challenge, it achieves a $7.2\times$ compression ratio with no accuracy loss and a $15.3\times$ compression ratio at 74.79% accuracy. Further analyses suggest that representation sharing can frequently happen across network layers and that learning shared representations for an entire DNN can achieve better accuracy at the same compression ratio than compressing the model as multiple separate parts. We release PyTorch code to facilitate DNN deployment on resource-constrained devices and spur future research on efficient representations and storage of DNN parameters.

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Categorical Reparameterization with Gumbel-Softmax
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Compressing Deep Convolutional Networks using Vector Quantization
• Rethinking the Value of Network Pruning
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Pruning Filters for Efficient ConvNets
• Binary Neural Networks: A Survey
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• The State of Sparsity in Deep Neural Networks
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Training deep neural networks with low precision multiplications
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Bringing AI To Edge: From Deep Learning's Perspective
• What is the State of Neural Network Pruning?
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Training wide residual networks for deployment using a single bit for each weight
• Compressing Word Embeddings via Deep Compositional Code Learning
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• Post-Training 4-bit Quantization on Embedding Tables
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?