Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

Abstract

The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on mobile devices. To this end, we proposed a simple yet effective scheme for compressing convolutions though applying k-means clustering on the weights, compression is achieved through weight-sharing, by only recording $K$ cluster centers and weight assignment indexes. We then introduced a novel spectrally relaxed $k$-means regularization, which tends to make hard assignments of convolutional layer weights to $K$ learned cluster centers during re-training. We additionally propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We finally evaluated Deep $k$-Means across several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss. The code is available at https://github.com/Sandbox3aster/Deep-K-Means

arXiv commentAccepted by ICML 2018

References in corpus (11)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Wide Residual Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Predicting Parameters in Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Convolutional neural networks with low-rank regularization
• Ultimate tensorization: compressing convolutional and FC layers alike
• The Power of Sparsity in Convolutional Neural Networks
• Structural Compression of Convolutional Neural Networks

Cited by in corpus (28)

• DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICs
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domain
• Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Neural Network Retraining for Model Serving
• DKM: Differentiable K-Means Clustering Layer for Neural Network Compression
• LegoNet: Memory Footprint Reduction Through Block Weight Clustering
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
• Approximations in Deep Learning
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators
• DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures
• DP-Net: Dynamic Programming Guided Deep Neural Network Compression
• Kernel Quantization for Efficient Network Compression
• Cluster Regularized Quantization for Deep Networks Compression
• Growing Deep Forests Efficiently with Soft Routing and Learned Connectivity
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training