IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
arXiv:2011.00677
Abstract
Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.
arXiv commentAccepted at COLING 2020 - The 28th International Conference on Computational Linguistics
References in corpus (13)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Cross-lingual Language Model Pretraining
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Enriching Word Vectors with Subword Information
- Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
- Text Summarization with Pretrained Encoders
- Neural Summarization by Extracting Sentences and Words
- XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
- 75 Languages, 1 Model: Parsing Universal Dependencies Universally
- Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing