Dynamical Isometry: The Missing Ingredient for Neural Network Pruning

Abstract

Several recent works [40, 24] observed an interesting phenomenon in neural network pruning: A larger finetuning learning rate can improve the final performance significantly. Unfortunately, the reason behind it remains elusive up to date. This paper is meant to explain it through the lens of dynamical isometry [42]. Specifically, we examine neural network pruning from an unusual perspective: pruning as initialization for finetuning, and ask whether the inherited weights serve as a good initialization for the finetuning? The insights from dynamical isometry suggest a negative answer. Despite its critical role, this issue has not been well-recognized by the community so far. In this paper, we will show the understanding of this problem is very important -- on top of explaining the aforementioned mystery about the larger finetuning rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a clearer theoretical understanding of pruning, resolving the problem can also bring us considerable performance benefits in practice.

arXiv comment8 pages, 2 figures, 7 tables

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The State of Sparsity in Deep Neural Networks
• Data-dependent Initializations of Convolutional Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Faster gaze prediction with dense networks and Fisher pruning
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• A Closer Look at Structured Pruning for Neural Network Compression
• Network Pruning That Matters: A Case Study on Retraining Variants