Distilling Task-Specific Knowledge from BERT into Simple Neural Networks

Abstract

In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These developments have led to the conviction that previous-generation, shallower neural networks for language understanding are obsolete. In this paper, however, we demonstrate that rudimentary, lightweight neural networks can still be made competitive without architecture changes, external training data, or additional input features. We propose to distill knowledge from BERT, a state-of-the-art language representation model, into a single-layer BiLSTM, as well as its siamese counterpart for sentence-pair tasks. Across multiple datasets in paraphrasing, natural language inference, and sentiment classification, we achieve comparable results with ELMo, while using roughly 100 times fewer parameters and 15 times less inference time.

8 pages, 2 figures; first three authors contributed equally

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• ADADELTA: An Adaptive Learning Rate Method
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Do Deep Nets Really Need to be Deep?
• Pruning Filters for Efficient ConvNets
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• A Convolutional Neural Network for Modelling Sentences
• Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
• Assessing BERT's Syntactic Abilities
• A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
• FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks

Cited by in corpus (89)

• Knowledge Distillation: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Deep Learning Based Text Classification: A Comprehensive Review
• Reducing Transformer Depth on Demand with Structured Dropout
• DocBERT: BERT for Document Classification
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• TinyBERT: Distilling BERT for Natural Language Understanding
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Structured Pruning of a BERT-based Question Answering Model
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Blockwise Self-Attention for Long Document Understanding
• I-BERT: Integer-only BERT Quantization
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Transferring Inductive Biases through Knowledge Distillation
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
• Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?
• Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
• Load What You Need: Smaller Versions of Multilingual BERT
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Squeezed Deep 6DoF Object Detection Using Knowledge Distillation
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings
• Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage
• Simplified TinyBERT: Knowledge Distillation for Document Retrieval
• Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads
• Improving Neural Machine Translation with Pre-trained Representation
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• Learning Effective Representations for Person-Job Fit by Feature Fusion
• Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention
• A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems
• Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
• ICS-Assist: Intelligent Customer Inquiry Resolution Recommendation in Online Customer Service for Large E-Commerce Businesses
• One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
• Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• Detecting Adverse Drug Reactions from Twitter through Domain-Specific Preprocessing and BERT Ensembling
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Selective Knowledge Distillation for Neural Machine Translation
• Attention Mechanism with Energy-Friendly Operations
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Distilling BERT for low complexity network training
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
• NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
• Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation
• Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
• MergeDistill: Merging Pre-trained Language Models using Distillation
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Learning to Augment for Data-Scarce Domain BERT Knowledge Distillation
• Adversarial Self-Supervised Data-Free Distillation for Text Classification
• Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method
• Confidence Preservation Property in Knowledge Distillation Abstractions
• RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models
• RefBERT: Compressing BERT by Referencing to Pre-computed Representations
• Elbert: Fast Albert with Confidence-Window Based Early Exit
• Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenches
• FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks
• Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
• Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network
• Distilling Linguistic Context for Language Model Compression
• Risk factor identification for incident heart failure using neural network distillation and variable selection
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques
• Deep Learning Meets Projective Clustering
• BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search