The Curious Case of Neural Text Degeneration

Abstract

Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

arXiv commentPublished in ICLR 2020

References in corpus (9)

• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Deep Reinforcement Learning for Dialogue Generation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Adversarial Learning for Neural Dialogue Generation
• On Accurate Evaluation of GANs for Language Generation
• Eval all, trust a few, do wrong to none: Comparing sentence generation models
• Evaluating Text GANs as Language Models
• The Neural Noisy Channel

Cited by in corpus (156)

• Language Models are Few-Shot Learners
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Practical Program Repair in the Era of Large Pre-trained Language Models
• Human heuristics for AI-generated language are flawed
• Towards a Human-like Open-Domain Chatbot
• Neural Text Generation with Unlikelihood Training
• A Taxonomy of Prompt Modifiers for Text-To-Image Generation
• A Simple Language Model for Task-Oriented Dialogue
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Data Augmentation using Pre-trained Transformer Models
• Pix2seq: A Language Modeling Framework for Object Detection
• Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Materials science in the era of large language models: a perspective
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
• Generative Data Augmentation for Commonsense Reasoning
• Defending Against Neural Fake News
• Video Background Music Generation with Controllable Music Transformer
• Large Language Models Can Be Strong Differentially Private Learners
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• PolyGen: An Autoregressive Generative Model of 3D Meshes
• MERLOT: Multimodal Neural Script Knowledge Models
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Compressive Transformers for Long-Range Sequence Modelling
• Explaining Question Answering Models through Text Generation
• PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
• Knowledge-Aware Language Model Pretraining
• EmpTransfo: A Multi-head Transformer Architecture for Creating Empathetic Dialog Systems
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Computer-Aided Design as Language
• Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech
• Learning to summarize from human feedback
• GPT-based Generation for Classical Chinese Poetry
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
• Uncertainty Estimation in Autoregressive Structured Prediction
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology
• Accelerating Transformer Inference for Translation via Parallel Decoding
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts
• QuestEval: Summarization Asks for Fact-based Evaluation
• Explanation as a Defense of Recommendation
• CALYPSO: LLMs as Dungeon Masters' Assistants
• Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
• Neural Language Generation: Formulation, Methods, and Evaluation
• Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
• Beyond the C: Retargetable Decompilation using Neural Machine Translation
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• Trading Off Diversity and Quality in Natural Language Generation
• Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
• RuCoLA: Russian Corpus of Linguistic Acceptability
• More but Correct: Generating Diversified and Entity-revised Medical Response
• SLGPT: Using Transfer Learning to Directly Generate Simulink Model Files and Find Bugs in the Simulink Toolchain
• Comparative Explanations of Recommendations
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling
• Finding Experts in Transformer Models
• Automatic Medical Report Generation: Methods and Applications
• EIGEN: Event Influence GENeration using Pre-trained Language Models
• End-to-End Trainable Non-Collaborative Dialog System
• Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamics
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Parallel Scheduled Sampling
• Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations
• INSPIRED: Toward Sociable Recommendation Dialog Systems
• Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer
• Analysis of diversity-accuracy tradeoff in image captioning
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
• Creative GANs for generating poems, lyrics, and metaphors
• Black Box Recursive Translations for Molecular Optimization
• STADEE: STAtistics-based DEEp Detection of Machine Generated Text
• The Impact of Expertise in the Loop for Exploring Machine Rationality
• Mixed-Lingual Pre-training for Cross-lingual Summarization
• Improved Natural Language Generation via Loss Truncation
• A Measure-Theoretic Characterization of Tight Language Models
• Learning to Paraphrase Sentences to Different Complexity Levels
• Exploring Diversity in Back Translation for Low-Resource Machine Translation
• The Piano Inpainting Application
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• Emotional Neural Language Generation Grounded in Situational Contexts
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Mirror: A Natural Language Interface for Data Querying, Summarization, and Visualization
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Differentiable Retrieval Augmentation via Generative Language Modeling for E-commerce Query Intent Classification
• AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models
• Learning the Simplicity of Scattering Amplitudes
• Table-to-Text Generation with Pretrained Diffusion Models
• An Exploratory Study on Perceptual Spaces of the Singing Voice
• Designing Precise and Robust Dialogue Response Evaluators
• Spectrogram Inpainting for Interactive Generation of Instrument Sounds
• A Large-Scale Chinese Short-Text Conversation Dataset
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation
• ChatGPT as a commenter to the news: can LLMs generate human-like opinions?
• An Effective, Robust and Fairness-aware Hate Speech Detection Framework
• Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task
• Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval
• Exploring the Effectiveness of Abstract Syntax Tree Patterns for Algorithm Recognition
• Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
• Multimodal End-to-End Sparse Model for Emotion Recognition
• AI as a Tool for Fair Journalism: Case Studies from Malta
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• Energy-Based Models with Applications to Speech and Language Processing
• Summary-Oriented Question Generation for Informational Queries
• Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
• An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models
• GraphPlan: Story Generation by Planning with Event Graph
• An Approach to Improve Robustness of NLP Systems against ASR Errors
• Holdout-Based Fidelity and Privacy Assessment of Mixed-Type Synthetic Data
• Reverse Engineering Configurations of Neural Text Generation Models
• Diversifying Dialogue Generation with Non-Conversational Text
• Unsupervised Natural Question Answering with a Small Model
• Collocation2Text: Controllable Text Generation from Guide Phrases in Russian
• NewsDialogues: Towards Proactive News Grounded Conversation
• Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models
• Cue-word Driven Neural Response Generation with a Shrinking Vocabulary
• Neural Language Modeling for Contextualized Temporal Graph Generation
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Long-range and hierarchical language predictions in brains and algorithms
• SentenceMIM: A Latent Variable Language Model
• "Nice Try, Kiddo": Investigating Ad Hominems in Dialogue Responses
• Language Model Evaluation in Open-ended Text Generation
• Which Discriminator for Cooperative Text Generation?
• Evaluating the Reliability of Self-Explanations in Large Language Models
• Relevance-Promoting Language Model for Short-Text Conversation
• Autoencoders as Tools for Program Synthesis
• What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
• Local Knowledge Powered Conversational Agents
• Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric
• Understanding Neural Abstractive Summarization Models via Uncertainty
• Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
• Fine-Grained Opinion Summarization with Minimal Supervision
• Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task
• Investigating Label Bias in Beam Search for Open-ended Text Generation
• CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding
• Navigating Human Language Models with Synthetic Agents
• Self-conditioning pre-trained language models
• To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs
• Boosting Summarization with Normalizing Flows and Aggressive Training
• CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation
• Injecting Hierarchy with U-Net Transformers
• Lightweight Decoding Strategies for Increasing Specificity
• SDA: Improving Text Generation with Self Data Augmentation
• Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention
• DIVE: Towards Descriptive and Diverse Visual Commonsense Generation
• Augmenting Machine Learning with Information Retrieval to Recommend Real Cloned Code Methods for Code Completion
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems