SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

Abstract

In this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• How multilingual is Multilingual BERT?
• A Survey of Code-switched Speech and Language Processing
• SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
• DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries
• Task-Specific Pre-Training and Cross Lingual Transfer for Code-Switched Data

Cited by in corpus (3)

• Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts
• Task-Specific Pre-Training and Cross Lingual Transfer for Code-Switched Data
• Developing Successful Shared Tasks on Offensive Language Identification for Dravidian Languages