Spanish Pre-trained BERT Model and Evaluation Data

Abstract

The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.

arXiv commentPublished as workshop paper at Practical ML for Developing Countries Workshop @ ICLR 2020

Cited by in corpus (11)

• MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain
• ColBERT: Using BERT Sentence Embedding in Parallel Neural Networks for Computational Humor
• Word differences in news media of lower and higher peace countries revealed by natural language processing and machine learning
• Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations
• Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
• Deep Natural Language Feature Learning for Interpretable Prediction
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion
• Tag and correct: high precision post-editing approach to correction of speech recognition errors
• Cross-domain Sentiment Classification in Spanish
• Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats