Proving the Lottery Ticket Hypothesis: Pruning is All You Need

Abstract

The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis (as was also conjectured in Ramanujan et al., 2019), showing that for every bounded distribution and every target network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.

References in corpus (4)

• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Limitations of Lazy Training of Two-layers Neural Networks

Cited by in corpus (10)

• Pruning via Iterative Ranking of Sensitivity Statistics
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• GANs Can Play Lottery Tickets Too
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• Finding Everything within Random Binary Networks
• Membership Inference Attacks on Lottery Ticket Networks
• A Probabilistic Approach to Neural Network Pruning