Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

Abstract

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

References in corpus (10)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• An Analysis of Deep Neural Network Models for Practical Applications
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Net2Net: Accelerating Learning via Knowledge Transfer
• Resiliency of Deep Neural Networks under Quantization

Cited by in corpus (64)

• Knowledge Distillation: A Survey
• Learned Step Size Quantization
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Applications and Techniques for Fast Machine Learning in Science
• SEED: Self-supervised Distillation For Visual Representation
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Up or Down? Adaptive Rounding for Post-Training Quantization
• QKD: Quantization-aware Knowledge Distillation
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• Towards Efficient Training for Neural Network Quantization
• Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Knowledge Consistency between Neural Networks and Beyond
• Towards Unified INT8 Training for Convolutional Neural Network
• Fighting Quantization Bias With Bias
• HAWQV3: Dyadic Neural Network Quantization
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Relaxed Quantization for Discretized Neural Networks
• I-BERT: Integer-only BERT Quantization
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• LIT: Block-wise Intermediate Representation Training for Model Compression
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Precision Highway for Ultra Low-Precision Quantization
• DistFL: Distribution-aware Federated Learning for Mobile Scenarios
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework
• Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation
• Towards Lossless Binary Convolutional Neural Networks Using Piecewise Approximation
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• Evolutionary Bin Packing for Memory-Efficient Dataflow Inference Acceleration on FPGA
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware
• Hessian-Aware Pruning and Optimal Neural Implant
• BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
• Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition
• Data Efficient Stagewise Knowledge Distillation
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
• Efficient non-uniform quantizer for quantized neural network targeting reconfigurable hardware
• Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network
• Low Precision Policy Distillation with Application to Low-Power, Real-time Sensation-Cognition-Action Loop with Neuromorphic Computing
• NASB: Neural Architecture Search for Binary Convolutional Neural Networks
• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model
• Knowledge Distillation via Instance-level Sequence Learning
• GDRQ: Group-based Distribution Reshaping for Quantization
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• Privileged Features Distillation at Taobao Recommendations
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• Light Multi-segment Activation for Model Compression
• Secure Your Ride: Real-time Matching Success Rate Prediction for Passenger-Driver Pairs
• Towards Data-Free Domain Generalization