Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

Abstract

The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collection of tickets, which is referred to as ``winning tickets'', in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In particular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as ``super tickets''. We further show that the phase transition is task and model dependent -- as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experiments on the GLUE benchmark show that the super tickets improve single task fine-tuning by $0.9$ points on BERT-base and $1.0$ points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning.

arXiv commentThe 59th annual meeting of the Association for Computational Linguistics (ACL 2021)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A large annotated corpus for learning natural language inference
• Reducing Transformer Depth on Demand with Structured Dropout
• Stabilizing the Lottery Ticket Hypothesis
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• When BERT Plays the Lottery, All Tickets Are Winning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Scheduled DropHead: A Regularization Method for Transformer Models