Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

Abstract

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MT-DNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to \emph{distill} knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7\% (1.5\% absolute improvement\footnote{ Based on the GLUE leaderboard at https://gluebenchmark.com/leaderboard as of April 1, 2019.}). The code and pre-trained models will be made publicly available at https://github.com/namisan/mt-dnn.

8 pages, 2 figures and 3 tables

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Natural Language Processing (almost) from Scratch
• A Survey on Multi-Task Learning
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Bayesian Dark Knowledge
• Net2Net: Accelerating Learning via Knowledge Transfer
• Multilingual Neural Machine Translation with Knowledge Distillation
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension

Cited by in corpus (47)

• Knowledge Distillation: A Survey
• Language Models are Few-Shot Learners
• Pre-trained Models for Natural Language Processing: A Survey
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Multi-Task Learning with Deep Neural Networks: A Survey
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• DIET: Lightweight Language Understanding for Dialogue Systems
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• Is Retriever Merely an Approximator of Reader?
• Knowledge Distillation from Internal Representations
• CORD19STS: COVID-19 Semantic Textual Similarity Dataset
• Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• Explaining Bayesian Neural Networks
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Transferring Inductive Biases through Knowledge Distillation
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Task Selection Policies for Multitask Learning
• The Inception Team at NSURL-2019 Task 8: Semantic Question Similarity in Arabic
• A Hybrid Neural Network Model for Commonsense Reasoning
• Multi-task Graph Convolutional Neural Network for Calcification Morphology and Distribution Analysis in Mammograms
• Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
• Small Towers Make Big Differences
• Multi-Task Self-Training for Learning General Representations
• Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models
• Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention
• Improved Synthetic Training for Reading Comprehension
• Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
• A Study of Non-autoregressive Model for Sequence Generation
• Aligning the Pretraining and Finetuning Objectives of Language Models
• Dual-View Distilled BERT for Sentence Embedding
• A Morpho-Syntactically Informed LSTM-CRF Model for Named Entity Recognition
• A distillation based approach for the diagnosis of diseases
• CUHK at SemEval-2020 Task 4: CommonSense Explanation, Reasoning and Prediction with Multi-task Learning
• BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching
• Industry Scale Semi-Supervised Learning for Natural Language Understanding
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression
• BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search