Network Pruning That Matters: A Case Study on Retraining Variants

Abstract

Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional fine-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance. In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage of a large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith et al., 2019). By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining - a detail often overlooked by practitioners during the implementation of network pruning. One-sentence Summary: We study the effective of different retraining mechanisms while doing pruning

arXiv commentAccepted at ICLR 2021 (Poster)

References in corpus (17)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The State of Sparsity in Deep Neural Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Rigging the Lottery: Making All Tickets Winners
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• Discovering Neural Wirings
• Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem

Cited by in corpus (3)

• Filter Pruning by Switching to Neighboring CNNs with Good Attributes
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)