Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Abstract

Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

arXiv commentAdded comparison to concurrent work

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Semi-supervised Sequence Learning
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Sequence-Level Knowledge Distillation
• Few-shot learning of neural networks from scratch by pseudo example optimization
• Transformer to CNN: Label-scarce distillation for efficient text classification
• Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System
• Attention-Guided Answer Distillation for Machine Reading Comprehension

Cited by in corpus (90)

• A Survey on Visual Transformer
• Knowledge Distillation: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Reducing Transformer Depth on Demand with Structured Dropout
• TinyBERT: Distilling BERT for Natural Language Understanding
• Training with Quantization Noise for Extreme Model Compression
• Measuring and Reducing Gendered Correlations in Pre-trained Models
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• SEED: Self-supervised Distillation For Visual Representation
• Structured Pruning of a BERT-based Question Answering Model
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• Learning from others' mistakes: Avoiding dataset biases without modeling them
• Zero-shot Node Classification with Decomposed Graph Prototype Network
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• A Survey on Contextualised Semantic Shift Detection
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• End-to-End Self-Debiasing Framework for Robust NLU Training
• Parameter-Efficient Transfer Learning with Diff Pruning
• Optimal Subarchitecture Extraction For BERT
• Incivility Detection in Open Source Code Review and Issue Discussions
• The MultiBERTs: BERT Reproductions for Robustness Analysis
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• I-BERT: Integer-only BERT Quantization
• Analyzing and Mitigating Interference in Neural Architecture Search
• Tiny Transformers for Environmental Sound Classification at the Edge
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient
• Rethinking embedding coupling in pre-trained language models
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• Sentence Embeddings using Supervised Contrastive Learning
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• Large-Scale News Classification using BERT Language Model: Spark NLP Approach
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation
• Efficient pre-training objectives for Transformers
• Grounding Representation Similarity with Statistical Testing
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
• ViBERTgrid: A Jointly Trained Multi-Modal 2D Document Representation for Key Information Extraction from Documents
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Demystifying BERT: Implications for Accelerator Design
• IrEne: Interpretable Energy Prediction for Transformers
• Is neural language acquisition similar to natural? A chronological probing study
• Improved Synthetic Training for Reading Comprehension
• Fine-grained Sentiment Controlled Text Generation
• Automatic Sexism Detection with Multilingual Transformer Models
• Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
• Follow Your Path: a Progressive Method for Knowledge Distillation
• Caregiver Talk Shapes Toddler Vision: A Computational Study of Dyadic Play
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Weakly-Supervised Open-Retrieval Conversational Question Answering
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation
• Leveraging Local Variation in Data: Sampling and Weighting Schemes for Supervised Deep Learning
• PAUSE: Positive and Annealed Unlabeled Sentence Embedding
• Lookup or Exploratory: What is Your Search Intent?
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Programming with Neural Surrogates of Programs
• D2S: Document-to-Slide Generation Via Query-Based Text Summarization
• Characterizing Abhorrent, Misinformative, and Mistargeted Content on YouTube
• Retraining DistilBERT for a Voice Shopping Assistant by Using Universal Dependencies
• Why Can You Lay Off Heads? Investigating How BERT Heads Transfer
• AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
• Frustratingly Simple Pretraining Alternatives to Masked Language Modeling
• Are Compressed Language Models Less Subgroup Robust?
• Debiasing Methods in Natural Language Understanding Make Bias More Accessible
• Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
• Distilling Linguistic Context for Language Model Compression
• Explore BiLSTM-CRF-Based Models for Open Relation Extraction
• Knowledge Distillation with Noisy Labels for Natural Language Understanding
• Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
• Influence Patterns for Explaining Information Flow in BERT
• Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• Context-Aware Transformer Transducer for Speech Recognition
• DNN-Based Semantic Model for Rescoring N-best Speech Recognition List
• Structural analysis of an all-purpose question answering model
• Coreference Augmentation for Multi-Domain Task-Oriented Dialogue State Tracking