MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
arXiv:1909.04761
Abstract
Pretrained language models are promising particularly for low-resource languages as they only require unlabelled data. However, training existing models requires huge amounts of compute, while pretrained cross-lingual models often underperform on low-resource languages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tune language models efficiently in their own language. In addition, we propose a zero-shot method using an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code.
Proceedings of EMNLP-IJCNLP 2019
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Cross-lingual Language Model Pretraining
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Pointer Sentinel Mixture Models
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
- Multilingual Neural Machine Translation With Soft Decoupled Encoding
- Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling