Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping

Abstract

Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains. However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses. Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling. In this work, we propose a method based on progressive layer dropping that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency. Extensive experiments on BERT show that the proposed method achieves a 24% time reduction on average per sample and allows the pre-training to be 2.5 times faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.

References in corpus (8)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Layer Normalization
• Deep Networks with Stochastic Depth
• TinyBERT: Distilling BERT for Natural Language Understanding
• Learning Deep Transformer Models for Machine Translation
• Cognitive Graph for Multi-Hop Reading Comprehension at Scale

Cited by in corpus (6)

• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• Data Quality Matters: Suicide Intention Detection on Social Media Posts Using RoBERTa-CNN
• Future Lens: Anticipating Subsequent Tokens from a Single Hidden State
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• LightSeq2: Accelerated Training for Transformer-based Models on GPUs