TinyBERT: Distilling BERT for Natural Language Understanding

Abstract

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.

Findings of EMNLP 2020; results have been updated; code and model: https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• FitNets: Hints for Thin Deep Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Q8BERT: Quantized 8Bit BERT
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Universal Transformers
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• A Tensorized Transformer for Language Modeling
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Structured Pruning of a BERT-based Question Answering Model
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Cited by in corpus (31)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• The Cost of Training NLP Models: A Concise Overview
• Pretrained Transformers for Text Ranking: BERT and Beyond
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Structured Pruning of a BERT-based Question Answering Model
• Pre-trained Summarization Distillation
• TURL: Table Understanding through Representation Learning
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
• MiniVLM: A Smaller and Faster Vision-Language Model
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails
• Pre-Trained Models: Past, Present and Future
• Real-Time Execution of Large-scale Language Models on Mobile
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• Adversarial NLI for Factual Correctness in Text Summarisation Models
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• Unsupervised Pre-training for Natural Language Generation: A Literature Review
• Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
• RefBERT: Compressing BERT by Referencing to Pre-computed Representations
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning
• Influence Patterns for Explaining Information Flow in BERT