Knowledge Distillation with Noisy Labels for Natural Language Understanding

Abstract

Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of research is the impact of noisy (corrupted) labels on KD. We present, to the best of our knowledge, the first study on KD with noisy labels in Natural Language Understanding (NLU). We document the scope of the problem and present two methods to mitigate the impact of label noise. Experiments on the GLUE benchmark show that our methods are effective even under high noise levels. Nevertheless, our results indicate that more research is necessary to cope with label noise under the KD.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• WebVision Database: Visual Learning and Understanding from Web Data
• Unsupervised Label Noise Modeling and Loss Correction
• A Survey of Label-noise Representation Learning: Past, Present and Future
• End-to-End Self-Debiasing Framework for Robust NLU Training
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Role-Wise Data Augmentation for Knowledge Distillation
• Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching