Data-Free Distillation of Language Model by Text-to-Text Transfer

Zheyuan Bai, Xinduo Liu, Hailin Hu +3

Data-Free Knowledge Distillation (DFKD) plays a vital role in compressing the model when original training data is unavailable. Previous works for DFKD in NLP mainly focus on disti…