Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?
arXiv:1905.10617
Abstract
Exposure bias has been regarded as a central problem for auto-regressive language models (LM). It claims that teacher forcing would cause the test-time generation to be incrementally distorted due to the training-generation discrepancy. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we focus on the task of open-ended language generation, propose metrics to quantify the impact of exposure bias in the aspects of quality, diversity, and consistency. Our key intuition is that if we feed ground-truth data prefixes (instead of prefixes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the training-generation discrepancy in the prefix is removed. Both automatic and human evaluations are conducted in our experiments. On the contrary to the popular belief in exposure bias, we find that the the distortion induced by the prefix discrepancy is limited, and does not seem to be incremental during the generation. Moreover, our analysis reveals an interesting self-recovery ability of the LM, which we hypothesize to be countering the harmful effects from exposure bias.
References in corpus (12)
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Recent Trends in Deep Learning Based Natural Language Processing
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
- Long Text Generation via Adversarial Training with Leaked Information
- Adversarial Ranking for Language Generation
- Adversarial Generation of Natural Language
- On Accurate Evaluation of GANs for Language Generation
- Trading Off Diversity and Quality in Natural Language Generation
- On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
- Rethinking Exposure Bias In Language Modeling