Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy

Abstract

Neural network pruning is a popular technique used to reduce the inference costs of modern, potentially overparameterized, networks. Starting from a pre-trained network, the process is as follows: remove redundant parameters, retrain, and repeat while maintaining the same test accuracy. The result is a model that is a fraction of the size of the original with comparable predictive performance (test accuracy). Here, we reassess and evaluate whether the use of test accuracy alone in the terminating condition is sufficient to ensure that the resulting model performs well across a wide spectrum of "harder" metrics such as generalization to out-of-distribution data and resilience to noise. Across evaluations on varying architectures and data sets, we find that pruned networks effectively approximate the unpruned model, however, the prune ratio at which pruned networks achieve commensurate performance varies significantly across tasks. These results call into question the extent of \emph{genuine} overparameterization in deep learning and raise concerns about the practicability of deploying pruned networks, specifically in the context of safety-critical systems, unless they are widely evaluated beyond test accuracy to reliably predict their performance. Our code is available at https://github.com/lucaslie/torchprune.

arXiv commentPublished in MLSys 2021

References in corpus (24)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Rethinking the Value of Network Pruning
• Pruning Filters for Efficient ConvNets
• The State of Sparsity in Deep Neural Networks
• Slimmable Neural Networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• What Do Compressed Deep Neural Networks Forget?
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• HYDRA: Pruning Adversarially Robust Neural Networks
• Towards Compact and Robust Deep Neural Networks
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Overinterpretation reveals image classification model pathologies
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators

Cited by in corpus (5)

• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Robustness in Compressed Neural Networks for Object Detection
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Model Fusion via Neuron Transplantation