GLGE: A New General Language Generation Evaluation Benchmark
arXiv:2011.11928
Abstract
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet (The source code and dataset are publicly available at https://github.com/microsoft/glge).
arXiv commentFindings of Association for Computational Linguistics. ACL 2021
References in corpus (8)
- Teaching Machines to Read and Comprehend
- BERTScore: Evaluating Text Generation with BERT
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
- The Natural Language Decathlon: Multitask Learning as Question Answering
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
- A BERT Baseline for the Natural Questions
Cited by in corpus (8)
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
- BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
- Improving Non-autoregressive Generation with Mixup Training
- ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
- EL-Attention: Memory Efficient Lossless Attention for Generation
- Problems and Countermeasures in Natural Language Processing Evaluation
- Prediction, Selection, and Generation: Exploration of Knowledge-Driven Conversation System