Comparing Rewinding and Fine-tuning in Neural Network Pruning

Abstract

Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this paper, we compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al., (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.

Cited by in corpus (69)

• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Rethinking Weight Decay For Efficient Neural Network Pruning
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Scaling Laws for Deep Learning
• DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Robust Lottery Tickets for Pre-trained Language Models
• On the Predictability of Pruning Across Scales
• Network Pruning That Matters: A Case Study on Retraining Variants
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• Efficient Neural Network Training via Forward and Backward Propagation Sparsification
• Joint PMD Tracking and Nonlinearity Compensation with Deep Neural Networks
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• Neural network relief: a pruning algorithm based on neural activity
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Exploring Lottery Ticket Hypothesis in Media Recommender Systems
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Revisiting Loss Modelling for Unstructured Pruning
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• Convolutional Neural Network Simplification with Progressive Retraining
• Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
• GANs Can Play Lottery Tickets Too
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weights
• ESPN: Extremely Sparse Pruned Networks
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• Initialization and Regularization of Factorized Neural Layers
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Neural Networks at a Fraction with Pruned Quaternions
• Going Beyond Classification Accuracy Metrics in Model Compression
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• Energy Consumption Analysis of pruned Semantic Segmentation Networks on an Embedded GPU
• Rethinking the Value of Transformer Components
• Spending Your Winning Lottery Better After Drawing It
• Search Spaces for Neural Model Training
• Prune Your Model Before Distill It
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
• Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?
• An Approximation Algorithm for Optimal Subarchitecture Extraction
• The Elastic Lottery Ticket Hypothesis
• A Study on Efficiency in Continual Learning Inspired by Human Learning
• Reproducibility Study: Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• RingCNN: Exploiting Algebraically-Sparse Ring Tensors for Energy-Efficient CNN-Based Computational Imaging
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• LEAN: graph-based pruning for convolutional neural networks by extracting longest chains
• Plant 'n' Seek: Can You Find the Winning Ticket?
• Sparsity-Control Ternary Weight Networks
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression