Language GANs Falling Short

Abstract

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort

References in corpus (20)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Level Training with Recurrent Neural Networks
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Pointer Sentinel Mixture Models
• A note on the evaluation of generative models
• MaskGAN: Better Text Generation via Filling in the______
• Mode Regularized Generative Adversarial Networks
• A Note on the Inception Score
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Long Text Generation via Adversarial Training with Leaked Information
• Adversarial Ranking for Language Generation
• Texygen: A Benchmarking Platform for Text Generation Models
• Adversarial Feature Matching for Text Generation
• On Accurate Evaluation of GANs for Language Generation
• DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
• Analyzing Uncertainty in Neural Machine Translation
• Neural Text Generation: Past, Present and Beyond
• Eval all, trust a few, do wrong to none: Comparing sentence generation models
• CoT: Cooperative Training for Generative Modeling of Discrete Data

Cited by in corpus (30)

• Evaluation of Text Generation: A Survey
• Clinically Accurate Chest X-Ray Report Generation
• Defending Against Neural Fake News
• Unifying Human and Statistical Evaluation for Natural Language Generation
• A Tutorial on Deep Latent Variable Models of Natural Language
• Evaluating Text GANs as Language Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• On the Weaknesses of Reinforcement Learning for Neural Machine Translation
• Discriminative Adversarial Search for Abstractive Summarization
• Improved Natural Language Generation via Loss Truncation
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generation
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• Sparse Text Generation
• Extractive Summary as Discrete Latent Variables
• Contextual Temperature for Language Modeling
• Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
• Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
• Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis
• Language Model Evaluation in Open-ended Text Generation
• Improving Text Generation with Student-Forcing Optimal Transport
• Meta-CoTGAN: A Meta Cooperative Training Paradigm for Improving Adversarial Text Generation
• Neural Machine Translation: A Review and Survey
• Toward Improving Coherence and Diversity of Slogan Generation
• Nested-Wasserstein Self-Imitation Learning for Sequence Generation
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• SDA: Improving Text Generation with Self Data Augmentation
• TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation
• Improving Adversarial Text Generation by Modeling the Distant Future
• The Detection of Distributional Discrepancy for Text Generation