Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
arXiv:2110.08419
Abstract
Recent work has focused on compressing pre-trained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.
Accepted by EACL 2023
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- Language Models are Few-Shot Learners
- Shortcut Learning in Deep Neural Networks
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Just Train Twice: Improving Group Robustness without Training Group Information
- Characterising Bias in Compressed Models
- The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
- Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression