Residual Energy-Based Models for Text
arXiv:2004.10188
Abstract
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations of these models be reliably distinguished from real text by statistical discriminators? We find experimentally that the answer is affirmative when we have access to the training data for the model, and guardedly affirmative even if we do not. This suggests that the auto-regressive models can be improved by incorporating the (globally normalized) discriminators into the generative process. We give a formalism for this using the Energy-Based Model framework, and show that it indeed improves the results of the generative models, measured both in terms of perplexity and in terms of human evaluation.
long journal version
References in corpus (14)
- Sequence to Sequence Learning with Neural Networks
- On the difficulty of training Recurrent Neural Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Skip-Thought Vectors
- A Theory of Generative ConvNet
- Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- Implicit Generation and Generalization in Energy-Based Models
- Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
- Unifying Human and Statistical Evaluation for Natural Language Generation
- Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
- Discriminator Rejection Sampling
- Trading Off Diversity and Quality in Natural Language Generation
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies