LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
arXiv:2103.06418
Abstract
The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these models are computationally intensive and difficult to be deployed on resource-restricted devices. In this paper, we propose a simple yet effective distillation method (LightMBERT) for transferring the cross-lingual generalization ability of the multilingual BERT to a small student model. The experiment results empirically demonstrate the efficiency and effectiveness of LightMBERT, which is significantly better than the baselines and performs comparable to the teacher mBERT.
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- On the Cross-lingual Transferability of Monolingual Representations
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Multilingual Alignment of Contextual Word Representations
- XNLI: Evaluating Cross-lingual Sentence Representations
- How multilingual is Multilingual BERT?