Sparse Transfer Learning via Winning Lottery Tickets

Abstract

The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with a small fraction of favorable weights. These weights retain their dominant status throughout training -- in a very real sense, this sub-network "won the lottery" during initialization. The authors find sub-networks via unstructured magnitude pruning with 85-95% of parameters removed that train to the same accuracy as the original network at a similar speed, which they call winning tickets. In this paper, we extend the Lottery Ticket Hypothesis to a variety of transfer learning tasks. We show that sparse sub-networks with approximately 90-95% of weights removed achieve (and often exceed) the accuracy of the original dense network in several realistic settings. We experimentally validate this by transferring the sparse representation found via pruning on CIFAR-10 to SmallNORB and FashionMNIST for object recognition tasks.

arXiv comment15 pages, 9 figures. Workshop on Learning Transferable Skills, 33rd Conference on Neural Information Processing Systems, Vancouver, Canada

References in corpus (9)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The State of Sparsity in Deep Neural Networks
• Regularization for Deep Learning: A Taxonomy
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Cited by in corpus (10)

• Winning the Lottery with Continuous Sparsification
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• How Well Do Sparse Imagenet Models Transfer?
• Data-dependent Pruning to find the Winning Lottery Ticket