Neural Language Generation: Formulation, Methods, and Evaluation

Abstract

Recent advances in neural network-based generative modeling have reignited the hopes in having computer systems capable of seamlessly conversing with humans and able to understand natural language. Neural architectures have been employed to generate text excerpts to various degrees of success, in a multitude of contexts and tasks that fulfil various user needs. Notably, high capacity deep learning models trained on large scale datasets demonstrate unparalleled abilities to learn patterns in the data even in the lack of explicit supervision signals, opening up a plethora of new possibilities regarding producing realistic and coherent texts. While the field of natural language generation is evolving rapidly, there are still many open challenges to address. In this survey we formally define and categorize the problem of natural language generation. We review particular application tasks that are instantiations of these general formulations, in which generating natural language is of practical importance. Next we include a comprehensive outline of methods and neural architectures employed for generating diverse texts. Nevertheless, there is no standard way to assess the quality of text produced by these generative models, which constitutes a serious bottleneck towards the progress of the field. To this end, we also review current approaches to evaluating natural language generation systems. We hope this survey will provide an informative overview of formulations, methods, and assessments of neural natural language generation.

70 pages

References in corpus (54)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Conditional Generative Adversarial Nets
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Scaling Laws for Neural Language Models
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Generating Long Sequences with Sparse Transformers
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Towards Principled Methods for Training Generative Adversarial Networks
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Reformer: The Efficient Transformer
• Neural Machine Translation in Linear Time
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Neural Text Generation with Unlikelihood Training
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Stand-Alone Self-Attention in Vision Models
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• The Evolved Transformer
• Faithful to the Original: Fact Aware Neural Abstractive Summarization
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Learning and Evaluating General Linguistic Intelligence
• Why We Need New Evaluation Metrics for NLG
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Pre-training via Paraphrasing
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
• Generating Wikipedia by Summarizing Long Sequences
• Context-aware Natural Language Generation with Recurrent Neural Networks
• Adversarial Evaluation of Dialogue Models
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Challenges in Data-to-Document Generation
• End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• Coherent Dialogue with Attention-based Language Models
• Unifying Human and Statistical Evaluation for Natural Language Generation
• An Architecture for Deep, Hierarchical Generative Models
• Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
• An Adversarial Approach to High-Quality, Sentiment-Controlled Neural Dialogue Generation
• Introducing MathQA -- A Math-Aware Question Answering System
• Controllable Sentence Simplification: Employing Syntactic and Lexical Constraints
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Learning to Predict Explainable Plots for Neural Story Generation
• Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
• Insertion-Deletion Transformer
• Natural Language Generation Using Reinforcement Learning with External Rewards
• Natural Language Generation Challenges for Explainable AI
• Towards Neural Language Evaluators
• Do Transformers Need Deep Long-Range Memory