Neural Text Generation with Unlikelihood Training

Abstract

Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Sean Welleck and Ilia Kulikov contributed equally

References in corpus (10)

• A Deep Reinforced Model for Abstractive Summarization
• The Curious Case of Neural Text Degeneration
• Sequence Level Training with Recurrent Neural Networks
• Pointer Sentinel Mixture Models
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Hierarchical Neural Story Generation
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Controllable Abstractive Summarization
• Retrieve and Refine: Improved Sequence Generation Models For Dialogue

Cited by in corpus (37)

• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Evaluation of Text Generation: A Survey
• ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
• Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
• Explaining Question Answering Models through Text Generation
• Learning to summarize from human feedback
• Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
• Neural Language Generation: Formulation, Methods, and Evaluation
• Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
• Trading Off Diversity and Quality in Natural Language Generation
• Comparative Explanations of Recommendations
• A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling
• Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamics
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Cascaded Text Generation with Markov Transformers
• TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
• Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
• Learning to Paraphrase Sentences to Different Complexity Levels
• Improved Natural Language Generation via Loss Truncation
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
• Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
• Multi-stage Pretraining for Abstractive Summarization
• Multi-scale Transformer Language Models
• Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
• Reverse Engineering Configurations of Neural Text Generation Models
• Language Model Evaluation in Open-ended Text Generation
• Collocation2Text: Controllable Text Generation from Guide Phrases in Russian
• SDA: Improving Text Generation with Self Data Augmentation
• Towards Diverse Paragraph Captioning for Untrimmed Videos
• Beyond [CLS] through Ranking by Generation
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• Augmenting Machine Learning with Information Retrieval to Recommend Real Cloned Code Methods for Code Completion