Neural Language Generation: Formulation, Methods, and Evaluation
arXiv:2007.15780
Abstract
Recent advances in neural network-based generative modeling have reignited the hopes in having computer systems capable of seamlessly conversing with humans and able to understand natural language. Neural architectures have been employed to generate text excerpts to various degrees of success, in a multitude of contexts and tasks that fulfil various user needs. Notably, high capacity deep learning models trained on large scale datasets demonstrate unparalleled abilities to learn patterns in the data even in the lack of explicit supervision signals, opening up a plethora of new possibilities regarding producing realistic and coherent texts. While the field of natural language generation is evolving rapidly, there are still many open challenges to address. In this survey we formally define and categorize the problem of natural language generation. We review particular application tasks that are instantiations of these general formulations, in which generating natural language is of practical importance. Next we include a comprehensive outline of methods and neural architectures employed for generating diverse texts. Nevertheless, there is no standard way to assess the quality of text produced by these generative models, which constitutes a serious bottleneck towards the progress of the field. To this end, we also review current approaches to evaluating natural language generation systems. We hope this survey will provide an informative overview of formulations, methods, and assessments of neural natural language generation.
70 pages
References in corpus (143)
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Conditional Generative Adversarial Nets
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Language Models are Few-Shot Learners
- Teaching Machines to Read and Comprehend
- Convolutional Sequence to Sequence Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- BERTScore: Evaluating Text Generation with BERT
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Scaling Laws for Neural Language Models
- A Neural Conversational Model
- Improved Training of Wasserstein GANs
- Tutorial on Variational Autoencoders
- A Deep Reinforced Model for Abstractive Summarization
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Enhanced LSTM for Natural Language Inference
- The Curious Case of Neural Text Degeneration
- VQA: Visual Question Answering
- Sequence Level Training with Recurrent Neural Networks
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- A Recurrent Latent Variable Model for Sequential Data
- A C-LSTM Neural Network for Text Classification
- Attention in Natural Language Processing
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Non-Autoregressive Neural Machine Translation
- Generating Long Sequences with Sparse Transformers
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Fine-Tuning Language Models from Human Preferences
- Variational Autoencoder for Deep Learning of Images, Labels and Captions
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Towards Principled Methods for Training Generative Adversarial Networks
- MaskGAN: Better Text Generation via Filling in the______
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Reformer: The Efficient Transformer
- Neural Machine Translation in Linear Time
- A Simple Method for Commonsense Reasoning
- Selective Encoding for Abstractive Sentence Summarization
- TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
- What do Neural Machine Translation Models Learn about Morphology?
- Big Bird: Transformers for Longer Sequences
- Neural Text Generation with Unlikelihood Training
- A Simple, Fast Diverse Decoding Algorithm for Neural Generation
- Stand-Alone Self-Attention in Vision Models
- Adversarial Learning for Neural Dialogue Generation
- GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
- Evaluation of Text Generation: A Survey
- The Evolved Transformer
- How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
- Order Matters: Sequence to sequence for sets
- Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
- Faithful to the Original: Fact Aware Neural Abstractive Summarization
- Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
- Demystifying MMD GANs
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- Why We Need New Evaluation Metrics for NLG
- Adversarial Ranking for Language Generation
- Learning and Evaluating General Linguistic Intelligence
- Hierarchical Neural Story Generation
- Toward Controlled Generation of Text
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- Sequence-to-Sequence Learning as Beam-Search Optimization
- Style Transfer as Unsupervised Machine Translation
- Boundary-Seeking Generative Adversarial Networks
- HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
- Mutual Information and Diverse Decoding Improve Neural Machine Translation
- deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
- Language Generation with Recurrent Generative Adversarial Networks without Pre-training
- Adversarial Training for Large Neural Language Models
- Pre-training via Paraphrasing
- Personalized Dialogue Generation with Diversified Traits
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
- Multiplicative LSTM for sequence modelling
- Content preserving text generation with attribute controls
- Language GANs Falling Short
- On Accurate Evaluation of GANs for Language Generation
- Generating Wikipedia by Summarizing Long Sequences
- Adversarial Text Generation via Feature-Mover's Distance
- Context-aware Natural Language Generation with Recurrent Neural Networks
- Controllable Sentence Simplification
- Adversarial Evaluation of Dialogue Models
- KERMIT: Generative Insertion-Based Modeling for Sequences
- Non-Monotonic Sequential Text Generation
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
- On Multiplicative Integration with Recurrent Neural Networks
- Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
- DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
- LSTM based Conversation Models
- Challenges in Data-to-Document Generation
- Reliable Fidelity and Diversity Metrics for Generative Models
- Improving Neural Language Modeling via Adversarial Training
- End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
- Semi-Autoregressive Training Improves Mask-Predict Decoding
- Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
- Coherent Dialogue with Attention-based Language Models
- PaperRobot: Incremental Draft Generation of Scientific Ideas
- Knowledge-Aware Language Model Pretraining
- The price of debiasing automatic metrics in natural language evaluation
- Unifying Human and Statistical Evaluation for Natural Language Generation
- A Tutorial on Deep Latent Variable Models of Natural Language
- An Architecture for Deep, Hierarchical Generative Models
- Backward and Forward Language Modeling for Constrained Sentence Generation
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
- Insertion-based Decoding with automatically Inferred Generation Order
- A Deep Memory-based Architecture for Sequence-to-Sequence Learning
- On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
- Global Encoding for Abstractive Summarization
- Recurrent Memory Networks for Language Modeling
- Affect-LM: A Neural Language Model for Customizable Affective Text Generation
- An Adversarial Approach to High-Quality, Sentiment-Controlled Neural Dialogue Generation
- Introducing MathQA -- A Math-Aware Question Answering System
- Natural Language to Structured Query Generation via Meta-Learning
- Zero-Shot Dialog Generation with Cross-Domain Latent Actions
- The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
- Controllable Sentence Simplification: Employing Syntactic and Lexical Constraints
- Loss Functions for Multiset Prediction
- Better Summarization Evaluation with Word Embeddings for ROUGE
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
- On the Weaknesses of Reinforcement Learning for Neural Machine Translation
- Learning to Predict Explainable Plots for Neural Story Generation
- POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
- Enabling Language Models to Fill in the Blanks
- Hierarchical Text Generation using an Outline
- Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
- Insertion-Deletion Transformer
- Simple and Effective Text Simplification Using Semantic and Neural Methods
- Natural Language Generation Challenges for Explainable AI
- Natural Language Generation Using Reinforcement Learning with External Rewards
- Towards Neural Language Evaluators
- Unsupervised Neural Text Simplification
- Do Transformers Need Deep Long-Range Memory