How many winning tickets are there in one DNN?

Abstract

The recent lottery ticket hypothesis proposes that there is one sub-network that matches the accuracy of the original network when trained in isolation. We show that instead each network contains several winning tickets, even if the initial weights are fixed. The resulting winning sub-networks are not instances of the same network under weight space symmetry, and show no overlap or correlation significantly larger than expected by chance. If randomness during training is decreased, overlaps higher than chance occur, even if the networks are trained on different tasks. We conclude that there is rather a distribution over capable sub-networks, as opposed to a single winning ticket.

arXiv comment17 pages, 15 figures, under submission

References in corpus (9)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Rethinking the Value of Network Pruning
• Matching Networks for One Shot Learning
• Stabilizing the Lottery Ticket Hypothesis
• Fixup Initialization: Residual Learning Without Normalization
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Critical Learning Periods in Deep Neural Networks
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• The Differentially Private Lottery Ticket Mechanism