Neural Text Generation: Past, Present and Beyond

Abstract

This paper presents a systematic survey on recent development of neural text generation models. Specifically, we start from recurrent neural network language models with the traditional maximum likelihood estimation training scheme and point out its shortcoming for text generation. We thus introduce the recently proposed methods for text generation based on reinforcement learning, re-parametrization tricks and generative adversarial nets (GAN) techniques. We compare different properties of these models and the corresponding techniques to handle their common problems such as gradient vanishing and generation diversity. Finally, we conduct a benchmarking experiment with different types of neural text generation models on two well-known datasets and discuss the empirical results along with the aforementioned model properties.

References in corpus (16)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Building Natural-Language Generation Systems
• Generative Adversarial Imitation Learning
• MaskGAN: Better Text Generation via Filling in the______
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Long Text Generation via Adversarial Training with Leaked Information
• Texygen: A Benchmarking Platform for Text Generation Models
• Toward Controlled Generation of Text
• Adversarial Feature Matching for Text Generation
• Adversarial Generation of Natural Language
• Adversarial Multi-task Learning for Text Classification
• DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
• Neural Text Generation: A Practical Guide

Cited by in corpus (18)

• Evaluation of Text Generation: A Survey
• Language GANs Falling Short
• On Accurate Evaluation of GANs for Language Generation
• Evaluating Text GANs as Language Models
• CoT: Cooperative Training for Generative Modeling of Discrete Data
• Adversarial System Variant Approximation to Quantify Process Model Generalization
• Semantic Bottleneck Scene Generation
• Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation
• Meta-CoTGAN: A Meta Cooperative Training Paradigm for Improving Adversarial Text Generation
• Modern Machine and Deep Learning Systems as a way to achieve Man-Computer Symbiosis
• A Differentially Private Multi-Output Deep Generative Networks Approach For Activity Diary Synthesis
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
• Joint Text and Label Generation for Spoken Language Understanding
• Generation of Synthetic Electronic Medical Record Text
• Melody-Conditioned Lyrics Generation with SeqGANs
• Adversarial Machine Learning in Text Analysis and Generation