CLUE: A Chinese Language Understanding Evaluation Benchmark

Abstract

The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com

Accepted by COLING2020; 10 pages, 4 figures

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• ERNIE: Enhanced Representation through Knowledge Integration
• Skip-Thought Vectors
• XNLI: Evaluating Cross-lingual Sentence Representations
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

Cited by in corpus (18)

• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis
• HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
• Density-Based Dynamic Curriculum Learning for Intent Detection
• Summarising Historical Text in Modern Languages
• AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
• Unsupervised Explanation Generation for Machine Reading Comprehension
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations
• HyperText: Endowing FastText with Hyperbolic Geometry