A Generalized Lottery Ticket Hypothesis
arXiv:2107.06825
Abstract
We introduce a generalization to the lottery ticket hypothesis in which the notion of "sparsity" is relaxed by choosing an arbitrary basis in the space of parameters. We present evidence that the original results reported for the canonical basis continue to hold in this broader setting. We describe how structured pruning methods, including pruning units or factorizing fully-connected layers into products of low-rank matrices, can be cast as particular instances of this "generalized" lottery ticket hypothesis. The investigations reported here are preliminary and are provided to encourage further research along this direction.
arXiv commentWorkshop on Sparsity in Neural Networks: Advancing Understanding and Practice (SNN'21). Updates: New curve on Figure 2(left) and discussion on Li et al
References in corpus (7)
- Scaling Laws for Neural Language Models
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Predicting Parameters in Deep Learning
- Learning Structured Sparsity in Deep Neural Networks
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
- Full deep neural network training on a pruned weight budget