Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models
arXiv:2010.03542
Abstract
This paper describes Galileo's performance in SemEval-2020 Task 12 on detecting and categorizing offensive language in social media. For Offensive Language Identification, we proposed a multi-lingual method using Pre-trained Language Models, ERNIE and XLM-R. For offensive language categorization, we proposed a knowledge distillation method trained on soft labels generated by several supervised models. Our team participated in all three sub-tasks. In Sub-task A - Offensive Language Identification, we ranked first in terms of average F1 scores in all languages. We are also the only team which ranked among the top three across all languages. We also took the first place in Sub-task B - Automatic Categorization of Offense Types and Sub-task C - Offence Target Identification.
arXiv comment8 pages, 2 figures, 6 tables. Accepted at Proceedings of 14th International Workshop on Semantic Evaluation (SemEval-2020)
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- ERNIE: Enhanced Representation through Knowledge Integration
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Arabic Offensive Language on Twitter: Analysis and Experiments