Language Model Evaluation in Open-ended Text Generation

Abstract

Although current state-of-the-art language models have achieved impressive results in numerous natural language processing tasks, still they could not solve the problem of producing repetitive, dull and sometimes inconsistent text in open-ended text generation. Studies often attribute this problem to the maximum likelihood training objective, and propose alternative approaches by using stochastic decoding methods or altering the training objective. However, there is still a lack of consistent evaluation metrics to directly compare the efficacy of these solutions. In this work, we study different evaluation metrics that have been proposed to evaluate quality, diversity and consistency of machine-generated text. From there, we propose a practical pipeline to evaluate language models in open-ended generation task, and research on how to improve the model's performance in all dimensions by leveraging different auxiliary training objectives.

References in corpus (8)

• Language Models are Few-Shot Learners
• Attention is not Explanation
• Pointer Sentinel Mixture Models
• Neural Text Generation with Unlikelihood Training
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Unifying Human and Statistical Evaluation for Natural Language Generation
• A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation