Model compression via distillation and quantization

Abstract

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

arXiv comment21 pages, published as a conference paper at ICLR2018

References in corpus (17)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• WaveNet: A Generative Model for Raw Audio
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Understanding deep learning requires rethinking generalization
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Trained Ternary Quantization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Learning Structured Sparsity in Deep Neural Networks
• Hardware-oriented Approximation of Convolutional Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Recurrent Neural Networks With Limited Numerical Precision
• Quantized Convolutional Neural Networks for Mobile Devices
• Simple and Efficient Learning using Privileged Information

Cited by in corpus (104)

• Knowledge Distillation: A Survey
• On the Opportunities and Risks of Foundation Models
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Learned Step Size Quantization
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• A Survey on Methods and Theories of Quantized Neural Networks
• Bringing AI To Edge: From Deep Learning's Perspective
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Knowledge Distillation in Deep Learning and its Applications
• Privacy-Preserving Machine Learning: Methods, Challenges and Directions
• Applications and Techniques for Fast Machine Learning in Science
• SEED: Self-supervised Distillation For Visual Representation
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• SLSNet: Skin lesion segmentation using a lightweight generative adversarial network
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• QKD: Quantization-aware Knowledge Distillation
• DSConv: Efficient Convolution Operator
• Split Computing for Complex Object Detectors: Challenges and Preliminary Results
• Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation
• StackRec: Efficient Training of Very Deep Sequential Recommender Models by Iterative Stacking
• Regularizing Activation Distribution for Training Binarized Deep Networks
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• Joint Neural Architecture Search and Quantization
• A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
• HAWQV3: Dyadic Neural Network Quantization
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• Relaxed Quantization for Discretized Neural Networks
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• I-BERT: Integer-only BERT Quantization
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Pre-Trained Models: Past, Present and Future
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• Ternary Compression for Communication-Efficient Federated Learning
• Knowledge Squeezed Adversarial Network Compression
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Secure Watermark for Deep Neural Networks with Multi-task Learning
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework
• TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domain
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• Convolutional Fully-Connected Capsule Network (CFC-CapsNet): A Novel and Fast Capsule Network
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• ESAI: Efficient Split Artificial Intelligence via Early Exiting Using Neural Architecture Search
• Neural Network Compression Via Sparse Optimization
• CDFI: Compression-Driven Network Design for Frame Interpolation
• Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Compressing Deep Neural Networks via Layer Fusion
• Hessian-Aware Pruning and Optimal Neural Implant
• Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning
• Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition
• Data Efficient Stagewise Knowledge Distillation
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• EBJR: Energy-Based Joint Reasoning for Adaptive Inference
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• Online Knowledge Distillation with Diverse Peers
• Detecting Dead Weights and Units in Neural Networks
• Efficient non-uniform quantizer for quantized neural network targeting reconfigurable hardware
• SAIA: Split Artificial Intelligence Architecture for Mobile Healthcare System
• Distillation Strategies for Proximal Policy Optimization
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• NASB: Neural Architecture Search for Binary Convolutional Neural Networks
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• NeRV: Neural Representations for Videos
• Student Network Learning via Evolutionary Knowledge Distillation
• YolactEdge: Real-time Instance Segmentation on the Edge
• Follow Your Path: a Progressive Method for Knowledge Distillation
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs
• High Frequency Residual Learning for Multi-Scale Image Classification
• ESPN: Extremely Sparse Pruned Networks
• Churn Reduction via Distillation
• Designing Interpretable Approximations to Deep Reinforcement Learning
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
• Abductive Reasoning as Self-Supervision for Common Sense Question Answering
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Tensor train decompositions on recurrent networks
• A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts
• Dynamic Multi-path Neural Network
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Towards Data-Free Domain Generalization
• Exploring Bit-Slice Sparsity in Deep Neural Networks for Efficient ReRAM-Based Deployment
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training
• Self-Supervised Encoder for Fault Prediction in Electrochemical Cells
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Multi-Scale Aligned Distillation for Low-Resolution Detection
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• Transferring Knowledge with Attention Distillation for Multi-Domain Image-to-Image Translation
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• AsymmNet: Towards ultralight convolution neural networks using asymmetrical bottlenecks
• Ship Detection: Parameter Server Variant