MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Abstract

Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its variants) have achieved remarkable success in varieties of NLP tasks. However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this work, we present a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation. The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher). Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student. Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys) that have been used in existing works. Moreover, we show that introducing a teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large pre-trained Transformer models. Experimental results demonstrate that our monolingual model outperforms state-of-the-art baselines in different parameter size of student models. In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model. We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models.

arXiv commentCode and models: https://github.com/microsoft/unilm/tree/master/minilm

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• A Simple Method for Commonsense Reasoning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• XNLI: Evaluating Cross-lingual Sentence Representations
• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Cited by in corpus (93)

• A Survey on Visual Transformer
• Knowledge Distillation: A Survey
• Pre-trained Models for Natural Language Processing: A Survey
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• Deep Learning Based Text Classification: A Comprehensive Review
• TinyBERT: Distilling BERT for Natural Language Understanding
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
• Pre-trained Summarization Distillation
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific Literature
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• Lightweight Transformers for Clinical Natural Language Processing
• Neural Retrievers are Biased Towards LLM-Generated Content
• Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
• MiniVLM: A Smaller and Faster Vision-Language Model
• Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
• Entity-aware Transformers for Entity Search
• Adversarial Robustness of Neural-Statistical Features in Detection of Generative Transformers
• Exploring Social Media for Early Detection of Depression in COVID-19 Patients
• Predicting Information Pathways Across Online Communities
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• Papeos: Augmenting Research Papers with Talk Videos
• An Argumentative Dialogue System for COVID-19 Vaccine Information
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• Human-interpretable clustering of short-text using large language models
• BEBERT: Efficient and Robust Binary Ensemble BERT
• I-BERT: Integer-only BERT Quantization
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Analyzing and Mitigating Interference in Neural Architecture Search
• Zero-shot Bilingual App Reviews Mining with Large Language Models
• Fast Vocabulary Transfer for Language Model Compression
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
• Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• Improving Non-autoregressive Generation with Mixup Training
• Large-Scale Knowledge Synthesis and Complex Information Retrieval from Biomedical Documents
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• Effective Inference-Free Retrieval for Learned Sparse Representations
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Quantifying the Dissimilarity of Texts
• CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation
• Simplified TinyBERT: Knowledge Distillation for Document Retrieval
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• RAFT: A Real-World Few-Shot Text Classification Benchmark
• A Neighbourhood Framework for Resource-Lean Content Flagging
• LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
• Knowledge Graph Based Repository-Level Code Generation
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• WikiMuTe: A web-sourced dataset of semantic descriptions for music audio
• DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning
• Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
• Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
• RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
• Enhancing Question Generation with Commonsense Knowledge
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
• Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots
• How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
• Exploring the Promises of Transformer-Based LMs for the Representation of Normative Claims in the Legal Domain
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
• NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application
• Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings
• Multi-word Tokenization for Sequence Compression
• Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• Learning to Augment for Data-Scarce Domain BERT Knowledge Distillation
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Hubness Reduction Improves Sentence-BERT Semantic Spaces
• RuSentEval: Linguistic Source, Encoder Force!
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• Distilling Linguistic Context for Language Model Compression
• GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models
• Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation
• ScopeIt: Scoping Task Relevant Sentences in Documents
• Dynamic Knowledge Distillation for Pre-trained Language Models
• Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation
• Anchor-based Plain Net for Mobile Image Super-Resolution
• Knowledge Distillation with Noisy Labels for Natural Language Understanding
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
• Student-Teacher Learning from Clean Inputs to Noisy Inputs
• Intent Classification and Slot Filling for Privacy Policies
• WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach