Deep Neural Compression Via Concurrent Pruning and Self-Distillation

Abstract

Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed {\em cross-correlation objective for self-distilled pruning} implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Estimating Mutual Information
• FitNets: Hints for Thin Deep Nets
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Variational Dropout Sparsifies Deep Neural Networks
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Self-Distillation Amplifies Regularization in Hilbert Space
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Contrastive Representation Distillation
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• Does Knowledge Distillation Really Work?
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning