Language Model Evaluation in Open-ended Text Generation

Abstract

Although current state-of-the-art language models have achieved impressive results in numerous natural language processing tasks, still they could not solve the problem of producing repetitive, dull and sometimes inconsistent text in open-ended text generation. Studies often attribute this problem to the maximum likelihood training objective, and propose alternative approaches by using stochastic decoding methods or altering the training objective. However, there is still a lack of consistent evaluation metrics to directly compare the efficacy of these solutions. In this work, we study different evaluation metrics that have been proposed to evaluate quality, diversity and consistency of machine-generated text. From there, we propose a practical pipeline to evaluate language models in open-ended generation task, and research on how to improve the model's performance in all dimensions by leveraging different auxiliary training objectives.

References in corpus (20)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Language Models are Few-Shot Learners
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• The Curious Case of Neural Text Degeneration
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• End-To-End Memory Networks
• Attention is not Explanation
• Pointer Sentinel Mixture Models
• Fine-Tuning Language Models from Human Preferences
• Neural Text Generation with Unlikelihood Training
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Language Models as Knowledge Bases?
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Language GANs Falling Short
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Eval all, trust a few, do wrong to none: Comparing sentence generation models
• A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation