And the Bit Goes Down: Revisiting the Quantization of Neural Networks

Abstract

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.

arXiv commentICLR 2020 camera-ready

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Towards Accurate Binary Convolutional Neural Network
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Data-Free Knowledge Distillation for Deep Neural Networks
• A Survey on Methods and Theories of Quantized Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Towards the Limit of Network Quantization
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• Training wide residual networks for deployment using a single bit for each weight
• Learning Discrete Weights Using the Local Reparameterization Trick

Cited by in corpus (16)

• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Training with Quantization Noise for Extreme Model Compression
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains
• Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Automated Design Space Exploration for optimised Deployment of DNN on Arm Cortex-A CPUs
• FleXOR: Trainable Fractional Quantization
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• LegoNet: Memory Footprint Reduction Through Block Weight Clustering
• Knowledge Distillation as Semiparametric Inference
• A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
• FrostNet: Towards Quantization-Aware Network Architecture Search
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization
• Accelerating Neural Network Inference by Overflow Aware Quantization
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression