EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets

Abstract

Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focusing on reducing inference time while still requiring an expensive training process. Other works use extremely large batch sizes to shorten the pre-training time, at the expense of higher computational resource demands. In this paper, inspired by the Early-Bird Lottery Tickets recently studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first to identify structured winning tickets in the early stage of BERT training. We apply those tickets towards efficient BERT training, and conduct comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks. Our results show that EarlyBERT achieves comparable performance to standard BERT, with 35~45% less training time. Code is available at https://github.com/VITA-Group/EarlyBERT.

arXiv commentAccepted at ACL-IJCNLP 2021

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Q8BERT: Quantized 8Bit BERT
• Reducing Transformer Depth on Demand with Structured Dropout
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Hopfield Networks is All You Need
• Structured Pruning of a BERT-based Question Answering Model
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• When BERT Plays the Lottery, All Tickets Are Winning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Contrastive Distillation on Intermediate Representations for Language Model Compression

Cited by in corpus (11)

• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• The Elastic Lottery Ticket Hypothesis
• Spending Your Winning Lottery Better After Drawing It
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation