Evaluating Text GANs as Language Models
arXiv:1810.12686
Abstract
Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of ``exposure bias''. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.
References in corpus (11)
- Sequence to Sequence Learning with Neural Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Character-Aware Neural Language Models
- Hierarchical Multiscale Recurrent Neural Networks
- Long Text Generation via Adversarial Training with Leaked Information
- Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
- Adversarial Generation of Natural Language
- Multiplicative LSTM for sequence modelling
- Language GANs Falling Short
- On Accurate Evaluation of GANs for Language Generation
- Neural Text Generation: Past, Present and Beyond
Cited by in corpus (8)
- The Curious Case of Neural Text Degeneration
- A Tutorial on Deep Latent Variable Models of Natural Language
- On the Weaknesses of Reinforcement Learning for Neural Machine Translation
- Multi-object Tracking via End-to-end Tracklet Searching and Ranking
- Rethinking Exposure Bias In Language Modeling
- ColdGANs: Taming Language GANs with Cautious Sampling Strategies
- Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
- Adversarial Machine Learning in Text Analysis and Generation