Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
arXiv:2106.09069
Abstract
Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an i.i.d. sample from the overall data, this approach overly simplifies the complexity of language and encourages overfitting to the head of the data distribution. As such, rare language phenomena or text about underrepresented groups are not equally included in the evaluation. To encourage more in-depth model analyses, researchers have proposed the use of multiple test sets, also called challenge sets, that assess specific capabilities of a model. In this paper, we develop a framework based on this idea which is able to generate controlled perturbations and identify subsets in text-to-scalar, text-to-text, or data-to-text settings. By applying this framework to the GEM generation benchmark, we propose an evaluation suite made of 80 challenge sets, demonstrate the kinds of analyses that it enables and shed light onto the limits of current generation models.
References in corpus (8)
- mT5: A massively multilingual pre-trained text-to-text transformer
- The Natural Language Decathlon: Multitask Learning as Question Answering
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- Evaluating Models' Local Decision Boundaries via Contrast Sets
- Text-to-Text Pre-Training for Data-to-Text Tasks
- ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models