How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding

Abstract

Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing (NLP) applications, little is understood about how one KD algorithm compares to another and whether these approaches can be complimentary to each other. In this work, we evaluate various KD algorithms on in-domain, out-of-domain and adversarial testing. We propose a framework to assess the adversarial robustness of multiple KD algorithms. Moreover, we introduce a new KD algorithm, Combined-KD, which takes advantage of two promising approaches (better training scheme and more efficient data augmentation). Our extensive experimental results show that Combined-KD achieves state-of-the-art results on the GLUE benchmark, out-of-domain generalization, and adversarial robustness compared to competitive methods.

arXiv commentAccepted as EMNLP 2021 Findings

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Synthetic and Natural Noise Both Break Neural Machine Translation
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT
• Robust Neural Machine Translation with Doubly Adversarial Inputs
• End-to-End Self-Debiasing Framework for Robust NLU Training
• Role-Wise Data Augmentation for Knowledge Distillation