Residual Energy-Based Models for Text Generation

Abstract

Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

published at ICLR 2020. arXiv admin note: substantial text overlap with arXiv:2004.10188

References in corpus (3)

• Implicit Generation and Generalization in Energy-Based Models
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Real or Fake? Learning to Discriminate Machine from Human Generated Text

Cited by in corpus (12)

• How to Train Your Energy-Based Models
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• A Distributional Approach to Controlled Text Generation
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Refining Deep Generative Models via Discriminator Gradient Flow
• Implicit Behavioral Cloning
• Residual Energy-Based Models for End-to-End Speech Recognition
• Unpaired Image-to-Image Translation via Latent Energy Transport
• Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Investigating Label Bias in Beam Search for Open-ended Text Generation
• Pre-Training Transformers as Energy-Based Cloze Models