A Gradient Flow Framework For Analyzing Network Pruning
arXiv:2009.11839
Abstract
Recent network pruning methods focus on pruning models early-on in training. To estimate the impact of removing a parameter, these methods use importance measures that were originally designed to prune trained models. Despite lacking justification for their use early-on in training, such measures result in surprisingly low accuracy loss. To better explain this behavior, we develop a general framework that uses gradient flow to unify state-of-the-art importance measures through the norm of model parameters. We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and is therefore appropriate for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models. We validate our claims on several VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100. Code available at https://github.com/EkdeepSLubana/flowandprune.
Accepted at ICLR, 2021
References in corpus (9)
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
- Pruning neural networks without any data by iteratively conserving synaptic flow
- Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- Generalization in Deep Networks: The Role of Distance from Initialization
- Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
- Rate Distortion For Model Compression: From Theory To Practice