Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
arXiv:2108.10640
Abstract
In this paper, we aim at improving Czech sentiment with transformer-based models and their multilingual versions. More concretely, we study the task of polarity detection for the Czech language on three sentiment polarity datasets. We fine-tune and perform experiments with five multilingual and three monolingual models. We compare the monolingual and multilingual models' performance, including comparison with the older approach based on recurrent neural networks. Furthermore, we test the multilingual models and their ability to transfer knowledge from English to Czech (and vice versa) with zero-shot cross-lingual classification. Our experiments show that the huge multilingual models can overcome the performance of the monolingual models. They are also able to detect polarity in another language without any training data, with performance not worse than 4.4 % compared to state-of-the-art monolingual trained models. Moreover, we achieved new state-of-the-art results on all three datasets.
arXiv commentAccepted to RANLP 2021, github: https://github.com/pauli31/improving-czech-sentiment-transformers
References in corpus (10)
- Decoupled Weight Decay Regularization
- CamemBERT: a Tasty French Language Model
- Reasoning about Entailment with Neural Attention
- Multilingual Denoising Pre-training for Neural Machine Translation
- mT5: A massively multilingual pre-trained text-to-text transformer
- Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
- BERTje: A Dutch BERT Model
- Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
- Multilingual is not enough: BERT for Finnish
- Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation