Statistical Mechanical Analysis of Neural Network Pruning

Abstract

Deep learning architectures with a huge number of parameters are often compressed using pruning techniques to ensure computational efficiency of inference during deployment. Despite multitude of empirical advances, there is a lack of theoretical understanding of the effectiveness of different pruning methods. We inspect different pruning techniques under the statistical mechanics formulation of a teacher-student framework and derive their generalization error (GE) bounds. It has been shown that Determinantal Point Process (DPP) based node pruning method is notably superior to competing approaches when tested on real datasets. Using GE bounds in the aforementioned setup we provide theoretical guarantees for their empirical observations. Another consistent finding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned) for a fixed number of parameters. We use our theoretical setup to prove this finding and show that even the baseline random edge pruning method performs better than the DPP node pruning method. We also validate this empirically on real datasets.

arXiv commentAuthors Ankani Chattoraj and Boyu Zhang made an equal contribution

References in corpus (11)

• A Survey on Deep Learning in Medical Image Analysis
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The State of Sparsity in Deep Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• Data-Independent Neural Pruning via Coresets
• Single Shot Structured Pruning Before Training