Compressing Deep Convolutional Networks using Vector Quantization

Abstract

Deep convolutional neural networks (CNN) has become the most promising method for object recognition, repeatedly demonstrating record breaking results for image classification and object detection in recent years. However, a very deep CNN generally involves many layers with millions of parameters, making the storage of the network model to be extremely large. This prohibits the usage of deep CNNs on resource limited hardware, especially cell phones or other embedded devices. In this paper, we tackle this model storage issue by investigating information theoretical vector quantization methods for compressing the parameters of CNNs. In particular, we have found in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods. Simply applying k-means clustering to the weights or conducting product quantization can lead to a very good balance between model size and recognition accuracy. For the 1000-category classification task in the ImageNet challenge, we are able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Predicting Parameters in Deep Learning
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Visualizing and Understanding Convolutional Networks
• Fast Training of Convolutional Networks through FFTs

Cited by in corpus (278)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Learning both Weights and Connections for Efficient Neural Networks
• Tensorizing Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Binary Neural Networks: A Survey
• Group Sparse Regularization for Deep Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Bayesian Compression for Deep Learning
• Recent Advances in Convolutional Neural Networks
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• MCUNet: Tiny Deep Learning on IoT Devices
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• A Survey on Methods and Theories of Quantized Neural Networks
• Deep Adaptive Feature Embedding with Local Sample Distributions for Person Re-identification
• Tiny Machine Learning: Progress and Futures
• JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Discrimination-aware Network Pruning for Deep Model Compression
• An Entropy-based Pruning Method for CNN Compression
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Dynamic Network Surgery for Efficient DNNs
• Exploring Sparsity in Recurrent Neural Networks
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• TinyBERT: Distilling BERT for Natural Language Understanding
• Edge Intelligence: Architectures, Challenges, and Applications
• Streaming keyword spotting on mobile devices
• Training with Quantization Noise for Extreme Model Compression
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• The Power of Sparsity in Convolutional Neural Networks
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Bayesian Bits: Unifying Quantization and Pruning
• Robust Ultra-wideband Range Error Mitigation with Deep Learning at the Edge
• Fast YOLO: A Fast You Only Look Once System for Real-time Embedded Object Detection in Video
• Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
• DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Effective Quantization Methods for Recurrent Neural Networks
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Towards the Limit of Network Quantization
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Automated Pruning for Deep Neural Network Compression
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• MEC: Memory-efficient Convolution for Deep Neural Network
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections
• High Performance Zero-Memory Overhead Direct Convolutions
• DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Accelerating Neural ODEs Using Model Order Reduction
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Quantized Convolutional Neural Networks for Mobile Devices
• Compressing Convolutional Neural Networks
• ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations
• Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• Benchmarking Edge Computing Devices for Grape Bunches and Trunks Detection using Accelerated Object Detection Single Shot MultiBox Deep Learning Models
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Wide Compression: Tensor Ring Nets
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Communication-Efficient Edge AI: Algorithms and Systems
• Recent Advances in Deep Learning for Object Detection
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• A Programmable Approach to Neural Network Compression
• ACDC: A Structured Efficient Linear Layer
• Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank
• Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
• Collaborative Execution of Deep Neural Networks on Internet of Things Devices
• Rate Distortion For Model Compression: From Theory To Practice
• Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms
• Classification of the Extremely Low Frequency Magnetic Field Radiation Measurement from the Laptop Computers
• Evolutionary Synthesis of Deep Neural Networks via Synaptic Cluster-driven Genetic Encoding
• Deep Fried Convnets
• Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Knowledge Projection for Deep Neural Networks
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Knowledge Distillation for End-to-End Person Search
• Single Shot Structured Pruning Before Training
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
• Shift-based Primitives for Efficient Convolutional Neural Networks
• VerIDeep: Verifying Integrity of Deep Neural Networks through Sensitive-Sample Fingerprinting
• A Highly Parallel FPGA Implementation of Sparse Neural Network Training
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• On Periodic Functions as Regularizers for Quantization of Neural Networks
• OpenEI: An Open Framework for Edge Intelligence
• Local Feature Detectors, Descriptors, and Image Representations: A Survey
• Intermittent Learning: On-Device Machine Learning on Intermittently Powered System
• Evolution in Groups: A deeper look at synaptic cluster driven evolution of deep neural networks
• LCNN: Lookup-based Convolutional Neural Network
• Compressing Neural Language Models by Sparse Word Representations
• A Lite Distributed Semantic Communication System for Internet of Things
• Training Sparse Neural Networks
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
• PruneNet: Channel Pruning via Global Importance
• AdaCompress: Adaptive Compression for Online Computer Vision Services
• ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels
• Structured Convolution Matrices for Energy-efficient Deep learning
• KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
• Feature Flow Regularization: Improving Structured Sparsity in Deep Neural Networks
• DeepFont: Identify Your Font from An Image
• TKD: Temporal Knowledge Distillation for Active Perception
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
• PTEENet: Post-Trained Early-Exit Neural Networks Augmentation for Inference Cost Optimization
• Training Bit Fully Convolutional Network for Fast Semantic Segmentation
• Dynamic Capacity Networks
• Faster Neural Network Training with Approximate Tensor Operations
• Sparse Neural Networks Topologies
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• A Generic Network Compression Framework for Sequential Recommender Systems
• Lightweight Convolutional Representations for On-Device Natural Language Processing
• Progressive Neural Networks for Image Classification
• Deploy Large-Scale Deep Neural Networks in Resource Constrained IoT Devices with Local Quantization Region
• Deep Learning Towards Mobile Applications
• To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference
• An Automotive Case Study on the Limits of Approximation for Object Detection
• Shapley Value as Principled Metric for Structured Network Pruning
• Learning Student Networks via Feature Embedding
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion
• Efficient Semantic Scene Completion Network with Spatial Group Convolution
• At-Scale Evaluation of Weight Clustering to Enable Energy-Efficient Object Detection
• Data-Free Learning of Student Networks
• A flexible, extensible software framework for model compression based on the LC algorithm
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamics
• LegoNet: Memory Footprint Reduction Through Block Weight Clustering
• End-to-End Learned Image Compression with Quantized Weights and Activations
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Data Efficient Stagewise Knowledge Distillation
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Compressing complex convolutional neural network based on an improved deep compression algorithm
• Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds
• 2-bit Model Compression of Deep Convolutional Neural Network on ASIC Engine for Image Retrieval
• Scalar Quantization as Sparse Least Square Optimization
• DARC: Differentiable ARchitecture Compression
• BitNet: Bit-Regularized Deep Neural Networks
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• Training for 'Unstable' CNN Accelerator:A Case Study on FPGA
• Cross-Layer Distillation with Semantic Calibration
• Predicting Terrain Mechanical Properties in Sight for Planetary Rovers with Semantic Clues
• A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
• Finding trainable sparse networks through Neural Tangent Transfer
• LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition
• Automatic Rank Selection for High-Speed Convolutional Neural Network
• FFT-Based Deep Learning Deployment in Embedded Systems
• Unsupervised Learning using Pretrained CNN and Associative Memory Bank
• Compression of Deep Neural Networks for Image Instance Retrieval
• How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Distilling portable Generative Adversarial Networks for Image Translation
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• MAFAT: Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
• Merging and Evolution: Improving Convolutional Neural Networks for Mobile Applications
• Compression of Deep Neural Networks on the Fly
• Parameter Efficient Deep Neural Networks with Bilinear Projections
• Network Automatic Pruning: Start NAP and Take a Nap
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Compressing LSTM Networks by Matrix Product Operators
• On the Uniqueness of Binary Quantizers for Maximizing Mutual Information
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
• Principal Component Networks: Parameter Reduction Early in Training
• Computation-Performance Optimization of Convolutional Neural Networks with Redundant Kernel Removal
• Table-Based Neural Units: Fully Quantizing Networks for Multiply-Free Inference
• Multi-modal Experts Network for Autonomous Driving
• Deep Neural Network Capacity
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• WSNet: Compact and Efficient Networks Through Weight Sampling
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Efficient Inferencing of Compressed Deep Neural Networks
• BasisConv: A method for compressed representation and learning in CNNs
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation
• Masked LARk: Masked Learning, Aggregation and Reporting worKflow
• TinyGAN: Distilling BigGAN for Conditional Image Generation
• Efficient Crowd Counting via Structured Knowledge Transfer
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Going Beyond Classification Accuracy Metrics in Model Compression
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• VeriDL: Integrity Verification of Outsourced Deep Learning Services (Extended Version)
• Active multi-fidelity Bayesian online changepoint detection
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks
• Learning Light-Weight Translation Models from Deep Transformer
• Channel Pruning Guided by Classification Loss and Feature Importance
• Domain Adaptation Regularization for Spectral Pruning
• Class-dependent Compression of Deep Neural Networks
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis
• Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Collaborative Inference for Efficient Remote Monitoring
• Information-Theoretic Understanding of Population Risk Improvement with Model Compression
• Training Quantized Neural Networks to Global Optimality via Semidefinite Programming
• CircConv: A Structured Convolution with Low Complexity
• Robust Student Network Learning
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• CompactNet: Platform-Aware Automatic Optimization for Convolutional Neural Networks
• Deep Learning Acceleration Techniques for Real Time Mobile Vision Applications
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Reducing the Model Order of Deep Neural Networks Using Information Theory
• Lattice Rescoring Strategies for Long Short Term Memory Language Models in Speech Recognition
• Reducing Inference Latency with Concurrent Architectures for Image Recognition
• Prune2Edge: A Multi-Phase Pruning Pipelines to Deep Ensemble Learning in IIoT
• End-to-end Learned Image Compression with Fixed Point Weight Quantization
• DP-Net: Dynamic Programming Guided Deep Neural Network Compression
• Constrained-size Tensorflow Models for YouTube-8M Video Understanding Challenge
• Binary Neural Network for Speaker Verification
• A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)
• (Pen-) Ultimate DNN Pruning
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Condensing Two-stage Detection with Automatic Object Key Part Discovery
• MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• PERMDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matrices
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training
• Multi-objective Evolutionary Approach for Efficient Kernel Size and Shape for CNN
• NodeDrop: A Condition for Reducing Network Size without Effect on Output
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• Elbert: Fast Albert with Confidence-Window Based Early Exit
• Fast Intent Classification for Spoken Language Understanding
• CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
• Distilling Knowledge From a Deep Pose Regressor Network
• Robust error bounds for quantised and pruned neural networks
• Kernel Quantization for Efficient Network Compression
• LEAN: graph-based pruning for convolutional neural networks by extracting longest chains
• A Greedy Algorithm for Quantizing Neural Networks
• An Embedded Deep Learning based Word Prediction
• Scalable Compression of Deep Neural Networks
• Block-term Tensor Neural Networks
• All Word Embeddings from One Embedding
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• Approximate Random Dropout
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• A Survey on Green Deep Learning
• Random matrices in service of ML footprint: ternary random features with no performance loss
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• Seeing Convolution Through the Eyes of Finite Transformation Semigroup Theory: An Abstract Algebraic Interpretation of Convolutional Neural Networks
• Virtual Experience to Real World Application: Sidewalk Obstacle Avoidance Using Reinforcement Learning for Visually Impaired