Problems and Countermeasures in Natural Language Processing Evaluation
arXiv:2104.09712
Abstract
Evaluation in natural language processing guides and promotes research on models and methods. In recent years, new evalua-tion data sets and evaluation tasks have been continuously proposed. At the same time, a series of problems exposed by ex-isting evaluation have also restricted the progress of natural language processing technology. Starting from the concept, com-position, development and meaning of natural language evaluation, this article classifies and summarizes the tasks and char-acteristics of mainstream natural language evaluation, and then summarizes the problems and causes of natural language pro-cessing evaluation. Finally, this article refers to the human language ability evaluation standard, puts forward the concept of human-like machine language ability evaluation, and proposes a series of basic principles and implementation ideas for hu-man-like machine language ability evaluation from the three aspects of reliability, difficulty and validity.
References in corpus (7)
- Teaching Machines to Read and Comprehend
- The Natural Language Decathlon: Multitask Learning as Question Answering
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
- GLGE: A New General Language Generation Evaluation Benchmark