Evaluation of contextual embeddings on less-resourced languages
arXiv:2107.10614
Abstract
The current dominance of deep neural networks in natural language processing is based on contextual embeddings such as ELMo, BERT, and BERT derivatives. Most existing work focuses on English; in contrast, we present here the first multilingual empirical comparison of two ELMo and several monolingual and multilingual BERT models using 14 tasks in nine languages. In monolingual settings, our analysis shows that monolingual BERT models generally dominate, with a few exceptions such as the dependency parsing task, where they are not competitive with ELMo models trained on large corpora. In cross-lingual settings, BERT models trained on only a few languages mostly do best, closely followed by massively multilingual BERT models.
arXiv comment45 pages
References in corpus (10)
- Distributed Representations of Words and Phrases and their Compositionality
- Language Models are Few-Shot Learners
- CamemBERT: a Tasty French Language Model
- Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
- Multilingual is not enough: BERT for Finnish
- Playing with Words at the National Library of Sweden -- Making a Swedish BERT
- Cross-lingual alignments of ELMo contextual embeddings
- FinEst BERT and CroSloEngual BERT: less is more in multilingual models
- Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
- EstBERT: A Pretrained Language-Specific BERT for Estonian