The Lottery Ticket Hypothesis for Pre-trained BERT Networks

Abstract

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at https://github.com/VITA-Group/BERT-Tickets.

arXiv commentNeurIPS 2020

References in corpus (21)

• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Q8BERT: Quantized 8Bit BERT
• The State of Sparsity in Deep Neural Networks
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• TinyBERT: Distilling BERT for Natural Language Understanding
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Structured Pruning of a BERT-based Question Answering Model
• The Early Phase of Neural Network Training
• Reweighted Proximal Pruning for Large-Scale Language Representation
• When BERT Plays the Lottery, All Tickets Are Winning
• Sparse Transfer Learning via Winning Lottery Tickets

Cited by in corpus (5)

• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• AUBER: Automated BERT Regularization
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
• The Elastic Lottery Ticket Hypothesis