Adversarial Self-Supervised Data-Free Distillation for Text Classification

Abstract

Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computationally expensive model to a resource-efficient lightweight model. However, most KD algorithms, especially in NLP, rely on the accessibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two-stage data-free distillation method, named Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed for compressing large-scale transformer-based models (e.g., BERT). To avoid text generation in discrete space, we introduce a Plug & Play Embedding Guessing method to craft pseudo embeddings from the teacher's hidden knowledge. Meanwhile, with a self-supervised module to quantify the student's ability, we adapt the difficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets.

11 pages, 5 figures, Accepted to EMNLP2020

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Q8BERT: Quantized 8Bit BERT
• A large annotated corpus for learning natural language inference
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Data-Free Knowledge Distillation for Deep Neural Networks
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Data-Free Adversarial Distillation
• Structured Pruning of a BERT-based Question Answering Model
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• Reweighted Proximal Pruning for Large-Scale Language Representation
• Transformer to CNN: Label-scarce distillation for efficient text classification
• Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings

Cited by in corpus (1)

• Contrastive Model Inversion for Data-Free Knowledge Distillation