Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning

Abstract

We analyse the pruning procedure behind the lottery ticket hypothesis arXiv:1803.03635v5, iterative magnitude pruning (IMP), when applied to linear models trained by gradient flow. We begin by presenting sufficient conditions on the statistical structure of the features under which IMP prunes those features that have smallest projection onto the data. Following this, we explore IMP as a method for sparse estimation.

Updated for Sparsity in Neural Networks Workshop

References in corpus (5)

• What is the State of Neural Network Pruning?
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Robust Pruning at Initialization

Cited by in corpus (6)

• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• GANs Can Play Lottery Tickets Too
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• A Probabilistic Approach to Neural Network Pruning