MergeDistill: Merging Pre-trained Language Models using Distillation

Abstract

Pre-trained multilingual language models (LMs) have achieved state-of-the-art results in cross-lingual transfer, but they often lead to an inequitable representation of languages due to limited capacity, skewed pre-training data, and sub-optimal vocabularies. This has prompted the creation of an ever-growing pre-trained model universe, where each model is trained on large amounts of language or domain specific data with a carefully curated, linguistically informed vocabulary. However, doing so brings us back full circle and prevents one from leveraging the benefits of multilinguality. To address the gaps at both ends of the spectrum, we propose MergeDistill, a framework to merge pre-trained LMs in a way that can best leverage their assets with minimal dependencies, using task-agnostic knowledge distillation. We demonstrate the applicability of our framework in a practical setting by leveraging pre-existing teacher LMs and training student LMs that perform competitively with or even outperform teacher LMs trained on several orders of magnitude more data and with a fixed model capacity. We also highlight the importance of teacher selection and its impact on student model performance.

ACL 2021 Findings

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• Multilingual is not enough: BERT for Finnish
• Distilling Large Language Models into Tiny and Effective Students using pQRNN