On Accurate Evaluation of GANs for Language Generation

Abstract

Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.

References in corpus (19)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Categorical Reparameterization with Gumbel-Softmax
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Improved Training of Wasserstein GANs
• Universal Sentence Encoder
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• MaskGAN: Better Text Generation via Filling in the______
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Demystifying MMD GANs
• Long Text Generation via Adversarial Training with Leaked Information
• Are GANs Created Equal? A Large-Scale Study
• DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
• Neural Text Generation: Past, Present and Beyond
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Eval all, trust a few, do wrong to none: Comparing sentence generation models

Cited by in corpus (29)

• The Curious Case of Neural Text Degeneration
• Evaluation of Text Generation: A Survey
• Classification Accuracy Score for Conditional Generative Models
• Language GANs Falling Short
• A Tutorial on Deep Latent Variable Models of Natural Language
• Fed-TGAN: Federated Learning Framework for Synthesizing Tabular Data
• Evaluating Text GANs as Language Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• Multi-object Tracking via End-to-end Tracklet Searching and Ranking
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Bilingual-GAN: A Step Towards Parallel Text Generation
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• CatGAN: Category-aware Generative Adversarial Networks with Hierarchical Evolutionary Learning for Category Text Generation
• ARAML: A Stable Adversarial Training Framework for Text Generation
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• Extractive Summary as Discrete Latent Variables
• Multi-scale Transformer Language Models
• Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation
• Adversarial Sub-sequence for Text Generation
• Collaborative Training of GANs in Continuous and Discrete Spaces for Text Generation
• Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling
• Distributional Discrepancy: A Metric for Unconditional Text Generation
• To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs
• The Detection of Distributional Discrepancy for Text Generation
• Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?
• Can adversarial training learn image captioning ?
• Joint Text and Label Generation for Spoken Language Understanding
• Fiction Sentence Expansion and Enhancement via Focused Objective and Novelty Curve Sampling