KLUE: Korean Language Understanding Evaluation

Abstract

We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.

arXiv comment76 pages, 10 figures, 36 tables

References in corpus (23)

• Decoupled Weight Decay Regularization
• Scaling Laws for Neural Language Models
• SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
• The Natural Language Decathlon: Multitask Learning as Question Answering
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Extracting Training Data from Large Language Models
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Building a Conversational Agent Overnight with Dialogue Self-Play
• XNLI: Evaluating Cross-lingual Sentence Representations
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
• What do Models Learn from Question Answering Datasets?
• KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension
• A Discourse-Level Named Entity Recognition and Relation Extraction Dataset for Chinese Literature Text
• KR-BERT: A Small-Scale Korean-Specific Language Model
• Open Korean Corpora: A Practical Report
• Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
• ParsiNLU: A Suite of Language Understanding Challenges for Persian

Cited by in corpus (7)

• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• USTC-NELSLIP at SemEval-2022 Task 11: Gazetteer-Adapted Integration Network for Multilingual Complex Named Entity Recognition
• Open Korean Corpora: A Practical Report
• Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
• Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• A Survey on Awesome Korean NLP Datasets