Robustness to Pruning Predicts Generalization in Deep Neural Networks

Abstract

Existing generalization measures that aim to capture a model's simplicity based on parameter counts or norms fail to explain generalization in overparameterized deep neural networks. In this paper, we introduce a new, theoretically motivated measure of a network's simplicity which we call prunability: the smallest \emph{fraction} of the network's parameters that can be kept while pruning without adversely affecting its training loss. We show that this measure is highly predictive of a model's generalization performance across a large set of convolutional networks trained on CIFAR-10, does not grow with network size unlike existing pruning-based measures, and exhibits high correlation with test set loss even in a particularly challenging double descent setting. Lastly, we show that the success of prunability cannot be explained by its relation to known complexity measures based on models' margin, flatness of minima and optimization speed, finding that our new measure is similar to -- but more predictive than -- existing flatness-based measures, and that its predictions exhibit low mutual information with those of other baselines.

References in corpus (10)

• Reconciling modern machine learning practice and the bias-variance trade-off
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• The State of Sparsity in Deep Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Spectrally-normalized margin bounds for neural networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Generalization in Deep Networks: The Role of Distance from Initialization
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

Cited by in corpus (5)

• Machine Learning and Deep Learning -- A review for Ecologists
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
• Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?